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‘Toward an Institute for Research in 
Communication Sciences 


LOTFI A. ZADEH 


Last June I attended in Moscow the 1959 meeting 
of the A. 8. Popov Society, which corresponds 
roughly to our IRE National Convention. I was 
very much impressed by the quality and breadth 
of Soviet work in information theory and related 
fields, and came back with the conviction that 
it is imperative to increase the quality and volume of 
our work in this area by a substantial margin. 

What can be done to strengthen our efforts? It 
seems to me that increasing the level of support 
for basic research in our universities and industrial 
laboratories would not by itself yield the desired 
results. Few of our professors can afford to con- 
centrate on research, most having to sandwich 
their research activities in between teaching, serving 
on committees, negotiating contracts, and super- 
vising M.S. and Ph.D. candidates. On the other 
hand, the atmosphere prevailing in most of our 
industrial laboratories is generally not conducive 
to the conduct of basic, nonprofit-oriented, long- 
term research. Indeed, it is a fact that much of the 
high quality basic work in our country is turned 
out by investigators who conduct research on an 
essentially full-time basis in institute-like organiza- 
tions within universities and industry. 

The key characteristics of such organizations are 
a) long-term, nonutilitarian attitude toward re- 
search, which implies a high degree of freedom for 
the researcher to work on problems of his own 
choosing, and insulation from pressures to produce 
results of immediate practical applicability, b) 
opportunity for interaction with other researchers, 
and c) high caliber of staff. A number of well-known 
institutions both in the United States and abroad 
have these characteristics, but they are embodied 
perhaps in their purest form in the Institute for 
Advanced Study, Princeton, N. J., which since its 
inception in 1930 has played a very significant role 
in the development of mathematics in this country. 

I should like to suggest that a comparable role 
in the development of communication theory and 
related fields could be played by an Institute 
for Research in Communication Sciences. Such an 
institute would not necessarily be patterned after 


the Institute for Advanced Study, nor after the 
Max Planck Institute in Germany, or the Institute 
for Automatics and Telemechanics in the Soviet 
Union. Rather it would be designed to meet the 
specific needs and interests of workers in the fields of 
information theory, communication theory, system 
theory, control theory, circuit theory, automata, 
biological systems, computation, machine transla- 
tion of languages, and related fields. It would be 
concerned with both theoretical and experimental 
research in these areas. It would not be affiliated 
with any one educational institution or industrial 
organization, although it might be advantageous to 
locate it in the vicinity of a university. It would 
have extensive library facilities and laboratories. It 
would be staffed primarily by temporary members, 
mostly on leave from industrial laboratories and 
universities. And it would be supported by the 
Federal government or a foundation, or a group 
of universities and industrial laboratories. A case 
in point, though not necessarily a model to follow 
closely, is the Brookhaven National Laboratory, 
Upton, Long Island. 

Such an institute would supplement, rather than 
supplant, the current research effort in universities 
and industry. It would provide a place where 
professors and industrial researchers could spend a 
year or two working full-time on their problems, 
interacting with one another in an atmosphere 
free from the usual pressures and harrassments. 
Perhaps this sounds like a dream, but it is a dream 
worth striving for. 

What are the chances of getting the necessary 
financial support for such an institute? Clearly, it 
would take a concerted effort of many people to 
implement the idea, and the prospects for success 
at this time are rather dim. Nonetheless, who- 
ever feels strongly that the creation of the Insti- 
tute for Research in Communication Sciences is 
in the national interest has the responsibility to 
press for its establishment regardless of how dif- 
ficult the task might be. History is replete with 
instances where eventual success has been achieved 
against even greater odds. 


IRE TRANSACTIONS ON INFORMATION THEORY 


Mar 


Optimum System Theory Using a General 


Bayes Criterion” 


V. S. PUGACHEVT 


Summary—An extension of the general method of obtaining 
an optimum system developed by the author! is given to include 
the case of nonlinear dependence of the observed function on 
signal parameters. The method affords effective determining of 
optimum systems designed for the detection and reproduction of 
signals in the presence of noise using various practically adequate 
criteria. 


I. FoRMULATION OF THE PROBLEM 


OST of the problems of obtaining an algorithm 
of an optimum system designed for detecting and 
reproducing signals in the presence of interfer- 

ences may be formulated as follows. 
For a random function 


Z(t) = g(t, U) + XY) (1) 


being observed in a certain domain 7' of the argument ¢, 
it is necessary to find an estimate W*(s) of a signal W(s) 
of the form 


Ws) = Ws, U, Y(s). (2) 


In (1) and (2) ¢ and y are definite functions, U is a 
random vector, and X(t) and Y(s) are random functions. 
Arguments t and s in problems of information theory 
and automatic control theory are the current time in 
the observation interval 7’, and the end of this interval, 
respectively. The domain 7’ of the variation of t, in which 
the function Z is observed, may be a discrete set of 
points, an interval s — 7 < t < s, or a set of nonover- 
lapping intervals. In other problems to which the theory 
developed below is applicable, arguments ¢ and s may 
be arbitrary scalar or vector variables varying over arbi- 
trary sets 7’ and S, respectively. 

The theory also includes the case in which the observed 
function Z(t) and the signal W(s) are vector random 
functions. To apply the theory to this case it is sufficient 
to consider the components Z,(#), W,(s) as scalar random 
functions of the arguments ft, s and the integers v, wu. 
For this reason the theory developed below is applicable 
as well to many other problems such as, for example, 
weather prediction, calculating the elements of the motion 
of satellites and cosmic rockets using observed data, etc. 


* Manuscript received by the PGIT, July 6, 1959. 

} Inst. of Automatics and Telemechanics, U.S.S.R. Academy of 
Sciences, Moscow. 

1'V. 8. Pugachev, “Determination of an optimum system using 
a general criterion,” Avtomat. i Telemekh., vol. 19, pp. 519-539; June, 
1958 (in Russian); also Automation and Remote Control, same issue 
(in English), 


Different criteria of optimality are adequate in differen 
practical problems. Yet most practically adequate criter 
belong to the class of Bayes criteria of the form 


MW, W*)) = 


where a loss function 1(W, W*) may be an, arbitra 
function or functional of the signal W and its estima 
W*, and the symbol M[X] denotes the mathematic 
expectation of the variable X.** The loss function 
may also depend upon certain indefinite parameters chose 
to satisfy the given complementary conditions. 

We shall obtain here the solution of the problem state 
above for any functions g, y and / under the followir 
general conditions: 


min, ( 


1) U is a finite-dimensional random vector with | 
known probability density f(w). 

2) [X(#), Y(s)] is a normally distributed random vect« 
function independent of the random vector U. 

3) For each realization z of the observed random fun 
tion Z there exists such a function w* that for ar 
function w the inequality 


MLW, w*) |z] < MLW, w) | 2] ( 


is valid. M[X |z] denotes here the condition 
expectation of X for a given realization z of . 
This condition is sufficient for the existence of suc 
a nonrandom operator A that the estimate 


W* = AZ ( 


provides the minimum average risk (3). This conc 

tion implies certain limitations upon the class 

admissible functions ¢, ~ and / and is satisfied 

almost all practical problems. | 
4) For all values of u the function g(t, u) is represen) 

able by the series of eigenfunctions of the rando 
function X(¢) in the domain 7’. | 
The realization of the random function Y is cor 
pletely determined if the corresponding realizatii 
of the random function X is known. 


Or 
nS" 


* A. Wald, “Statistical Decision Functions,” John Wiley a 
Sons, Inc., New York, N. Y.; 1950. 
3D. Van Meter, and D. Middleton, ‘Modern statistical < 
proaches to reception in communication theory,’”? IRE Trai 
oN INrorMATION THEORY, vol. IT-4, pp. 119-145; September, 19) 
* D. Middleton, and D. Van Meter, “Detection and extracti 
of signals in noise from the point of view of statistical decis’ 
theory, Parts I and II,” SIAM J., vol. 3, pp. 192-253, Decemk 
1955 and vol. 4, pp. 86-119; June, 1956. | 
|| 
| 
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Il. THe SoLutTion or THE PROBLEM 


To solve the problem we represent the vector random 
unction [X(#), Y(s)] by some canonical expansion*”” 


AW) — x Viz), Y= ye Viys), (6) 


vhere V, are uncorrelated random variables and z,/(t), 
,(s) are nonrandom coordinate functions. The expansions 
(6) may always be found in an infinite variety of ways 
or any finite domains 7, S(t ¢ 7, s ¢ S). In particular, 
eal random functions X, Y are always representable by 
‘xpansions (6) with real coordinate functions z,, y, and 
oefficients V,. Once real expansions (6) are obtained, 
t is possible to find a system of real linear functionals 
2 acting upon the values of functions of the variable 
varying in the domain 7’, satisfying 


Or, ae Or) (7) 


| 1 ee 
| x,(t) = ie Q, EE, %)), (8) 


where D, is the variance of the random variable V,, K, 
s the correlation function (covariance) of the random 
‘unction X and 4,, is Kronecker’s delta. Then the random 
coefficients V, contained in the expansion (6) of & are 
-epresentable by’’® 


! V, = OX. (9) 

In the case of a discrete observation domain 7’, the 
‘unctionals 9” are linear combinations of values of corres- 
sonding functions in points of the domain 7: 


tke 


4 acy (tr) « (10) 


. 
In the case of a continuous observation. domain 7 and 
scalar function Z, the functionals 2° are integrals ex- 
ending over the domain 7’: 

| 


Oy = D a,(t)y(1) di. (11) 


Finally in the case of a continuous observation domain 7 
nd vector function Z, the functionals 2” are sums of 
ntegrals: 


97 = Df an(Dr) at. (12) 


5V. §. Pugachey, “Use of canonical expansions of random 
unctions to derive optimal linear systems,’’ Aviomat. 1 Telemekh., 
ol. 17, pp. 489-499; June, 1956 (in Russian); also Automation 
md Remote Control, same issue (in English). - : 

6 V.S. Pugachev, “Theory of Random Functions and its Applica- 
ion to Automatic Control Problems,’’ State Publishing House of 
Pheoretical-Technical Literature, Moscow; 1957. (In Russian. ) 

7 Expectations of random functions X, Y may clearly be as- 
umed without loss of generality to be equal identically to zero. 
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Assumption 4) of the preceding section means that for 
all values of the vector wu the function ¢g(¢, uw) may be 
represented in the domain 7’ by the expansion 

g(t, u) = >> D,a,(u)z,(1) 


(te T). (13) 


where 


Dia) = 0 eG, u). (14) 


Assumption 5) of Section I means that the expansion 
(6) of the random function Y contains only such random 
variables V, as are also present in the expansion (6) 
of the random function X. That means also that all 
random variables V, are representable by (9). 

Using the fact that V, are independent normally distri- 
buted random variables, being linear transforms of the 
normally distributed random function X, we obtain the 
following expression for the conditional expectation of the 
loss function / with respect to the observed function Z: 


MLW, W) |Z) =¢f UvG,u, AZ — ao), W*) 


‘f(u) exp {LZ — 26(u)} du, (15) 
where A“ and L(u) are linear operators defined by 
AS 7&, Dp en EO) a Lao, (16) 
w(u) and 8(w) are functions defined by 
wou) = 2) Da(wyl(s) = A elt, wo), 
i Pieri) 


BU) = Ps D,o,(u) = Lw).e(t, u) 


and ¢ is a certain constant depending on the observed 
function Z and independent of W*. 

Thus the problem of finding an optimal estimate W* 
of the signal W is reduced to minimization of the integral 
(15) after calculating linear operators A“ and L(u) and 
functions w(u) and B(u). 

It is easily verified by direct substitution that linear 
operators A“ and L(u) satisfy 


AWK, @) = KuG, 
L(u) ,K,(E, 7) az g(t, u) (t & ri), 


(18) 


where, in addition to previous notations, K,,(s, ¢) is the 
cross-correlation function of random functions Y and X. 
Thus series (16) determine solutions of (18).°’° If (18) 
may be solved in finite form by some procedure, then 
operators A“, L(u) and functions w(u), B(u) may be 
found without using infinite series. 

In the case of a discrete observation domain 7’, (18) 
are systems of linear algebraic equations. In the case 
of a continuous observation domain 7’, (18) are Fredholm 
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integral equations of the first kind if Z is a scalar function, 
and systems of Fredholm integral equations of the first 
kind if Z is a vector function. To solve (18) in finite form 
the method of integral canonical representations of random 
functions may be used.*”® 

The sufficient conditions for convergence in mean square 
sense of the series 


eye 2 and > aworZ (19) 
are the existence of finite variance of the random function 
Y and the convergence for all w of the second series 
CTs 

Minimum conditional expectations (15) corresponding 
to successive partial sums of series (16) will converge in 
probability to the minimum value of the expectation 
M{l(W, W*) | Z] in this case for a wide class of functions 
l,¢ and y. 

The method of obtaining the optimum system algorithm, 
developed above, is also applicable to the case when some 
components (or all) of the vector U are nonrandom un- 
known quantities which may assume any values. In this 
case f(u) represents the joint probability density of random 
components of the vector U (unity if all components of 
the vector U are nonrandom). 


III. Some AppLicATIONS OF THE METHOD 
Defining the loss function J by 
0 if w= 
date se Oy wise, 


OS 0r <C. Or, hoe Oe sc 


l(w;.w*) = (20) 


Nit: = OF ws Se; 


the method developed in the preceding section yields the 
general solution of the problem of detecting a signal in 
noise. The constant \ is equal to unity if the ideal observer 
criterion is used, and is determined by a given value of 
the probability of a false alarm if the Neyman-Pearson 
criterion is used. Yet the loss function is reduced to a 
form (20) in the case of arbitrary chosen weights of 
correct decisions and errors of the two types. 


If ¢ is a linear function of components U,, --: , Uy 
of the vector U, 
N 
g(t, U) = > Unertt), (21) 
=1 


the method yields an explicit expression of the optimum 
system operator.’ 


8V. S. Pugachev, “Integral canonic representation of random 
functions and their application in deriving optimal linear systems,”’ 
Avtomat. 1 Telemekh., vol. 18, pp. 971-984; November, 1957 (in 
Russian), also Automation and Remote Control, same issue (in 
English). 

* V.S. Pugachev, ‘Method of solving the basic integral equation 
of the statistical theory of optimum systems in closed form,”’ 
Prikladnaia Matemat. 1 Mekh., vol. 23, pp. 3-14; January, 1959. 
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. The function y being also linear in U,, --- , Uy, Y 
random components of the vector U being normally 
distributed and the loss function depending only on th 
difference W* — W, the method yields a linear optimun 
system and the form of the loss function affects only < 
systematic part of the optimum system output, independ 
ent of the specific realization of the input Z. If, in addition 
the loss function / is a nondecreasing function of th 
modulus of the difference W* — W, then the optimun 
system is completely independent of the specific choice 
of the loss function 1 of that form and coincides witl 
that determined by the least mean square criterion fo: 
this case.’ These statements are also valid in the case 
when the observation domain T' is the interval s. — T < 
t < s and the loss function / is a functional of the form 


uw, W*) = i: 5G. eat ae 
S 


where | 
(23 


o is any function of the arguments indicated, and S i 
any domain of the variation of the argument s. 

In the case where Z(t) is a scalar function, U is ¢ 
nonrandom vector, the observation domain 7 is the 
intervals — T < ¢ < s, W = U, and the loss functior 
is determined by 


A = AG) = W*(s) — Wis), 


l(w, w*) = const — d(w* — w) (24 
where 6 is the Dirac delta function, the general techni 
ques derived in the preceding section yield the equation: 
determining the estimates of signal parameters obtainec 
by Slepian by application of the usual maximum likeli 
hood method.” 

The method may also be applied to find an estimat 
of an infinite-dimensional normally distributed vector U 
using the maximum likelihood criterion. There results i 
this case the infinite system of equations determinins 
estimates of components of the vector U, similar t 
Slepian’s system of equations. Using this fact, it is pos 
sible to obtain an algorithm for obtaining the maximun 
likelihood estimate of a normally distributed random fune 
tion W(t), upon which the observed function Z depends 


Z(t) = ult, WW) + X(). Ci 


For this purpose it is sufficient to express the randor, 
function W(#) by some canonical expansion: 


Wit) = x U,2w,(t). (26 


Then, departing from the equations, determining estimate 
of independent random variables U;, the system of equs 


0D, Slepian, “Estimation of signal parameters in the presenc 
of noise,” IRE Trans. on InrorMATION THEORY, vol. IT-3, p] 
68-89; March, 1954. ; 


| 
| 


960 


ons for the maximum likelihood estimate of the random 
ietion, W(t), which was previously obtained by Youla 
lows." There also results another system of equations 
termining that estimate of the random function W(é) 
‘hich is equivalent to Youla’s system of equations. 

|The general method discussed in the preceding section 
lso yields the solution of many other problems of optimum 
ystem theory and similar problems of many other 
ranches of science. 


IV. ConcLtupInGc REMARKS 


We see that the general method presented above yields 
he effective solution of various problems of applied sta- 
istical decision theory, particularly, of the statistical 
eory of signal detection and reproduction. 

The algorithm given by the method for obtaining a 


1 PD. Youla, “The use of the method of maximum likelihood in 
stimating continuous-modulated intelligence which has been 
orrupted by noise,’ IRE Trans. on INFORMATION THEORY, 
ol. IT-3, pp. 90-105; March, 1954. 
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signal estimate may be used as a base for real system 
design. The main difficulty which arises in the practical 
realization of such systems is generally the absence of 
necessary data characterizing the a priori distribution of 
the signal parameter U [7.e., the probability density f(u)]. 
To avoid this difficulty, the same algorithm may be 
applied to obtain the signal parameter U estimate in 
each cycle of the system acting, and to construct for 
each cycle an estimate for f(w) using estimates of U 
obtained in all previous cycles. Using such estimates of 
f(u) for each cycle of the system acting, we obtain a 
“self-learning”? system which will be nearer and nearer 
with each new cycle to the true optimum system cor- 
responding to the true probability distribution of the 
vector U.” 


2 K. Winkelbauer, ‘Experience in Games of Strategy and In 
Statistical Decision,” Trans. of the First Prague Conf. on Information 
Theory, Statistical Decision Functions and Random Processes, Liblice, 
November 28-30, 1956, Czech. Acad. of Sc., Prague, pp. 297-354, 
1957. 


Quantizing for Minimum Distortion* 


JOEL MAX} 


Summary—This paper discusses the problem of the minimization 
f the distortion of a signal by a quantizer when the number of 
utput levels of the quantizer is fixed. The distortion is defined as 
he expected value of some function of the error between the input 
nd the output of the quantizer. Equations are derived for the param- 
ters of a quantizer with minimum distortion. The equations are 
ot soluble without recourse to numerical methods, so an algorithm 
; developed to simplify their numerical solution. The case of an 
put signal with normally distributed amplitude and an expected 
quared error distortion measure is explicitly computed and values 
f the optimum quantizer parameters are tabulated. The optimization 
f a quantizer subject to the restriction that both input and output 
svels be equally spaced is also treated, and appropriate parameters 
re tabulated for the same case as above. 


* Manuscript received by the PGIT, September 25, 1959. 
‘his work was performed by the Lincoln Lab., Mass. Inst. Tech., 
exington, Mass., with the joint support of the U. S. Army, Navy, 
nd Air Force. j 

+ Lincoln Lab., Mass. Inst. Tech., Lexington, Mass. 


N MANY data-transmission systems, analog input 

signals are first converted to digital form at the 

transmitter, transmitted in digital form, and finally 
reconstituted at the receiver as analog signals. The result- 
ing output normally resembles the input signal but is 
not precisely the same since the quantizer at the trans- 
mitter produces the same digits for all Input amplitudes 
which lie in each of a finite number of amplitude ranges. 
The receiver must assign to each combination of digits 
a single value which will be the amplitude of the reconsti- 
tuted signal for an original input anywhere within the 
quantized range. The difference between input and output 
signals, assuming errorless transmission of the digits, is 
the quantization error. Since the digital transmission rate 
of any system is finite, one has to use a quantizer which 
sorts the input into a finite number of ranges, NV. For a 
given N, the system is described by specifying the end 
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points, v,, of the N input ranges, and an output level, y;, 
corresponding to each input range. If the amplitude 
probability density of the signal which is the quantizer 
input is given, then the quantizer output is a quantity 
whose amplitude probability density may easily be deter- 
mined as a function of the x,’s and y;,’s. Often it is appro- 
priate to define a distortion measure for the quantization 
process, which will be some statistic of the quantization 
error. Then one would like to choose the N y;’s and the 
associated 2,’s so as to minimize the distortion. If we 
define the distortion, D, as the expected value of f(e), 
where f is some function (differentiable), and ¢ is the 
quantization error, and call the input amplitude prob- 
ability density p(x), then 


D = Elfin — Sour) 


= = / ve f(a — yi)p(a) dx 


where ty11 = ©, 2; = — ©, and the convention is that 
an input between x, and z;,, has a corresponding output y;. 

If we wish to minimize D for fixed NV, we get necessary 
conditions by differentiating D with respect to the 2,’s 
and y;’s and setting derivatives equal to zero: 


is =f; — ¥:)pe;) — 1a; — yi)pe;) = 0 
] = Ph masts nN (1) 
gD = — [pe — yp) de = 0 
‘jane ig Vi (2) 
(1) becomes (for p(x;) # 0) 
f(a; — ¥;-1) = f(a; = Y;) f =e 2M (3) 
(2) becomes 
ae f(a — y;)p(x) dx = 0 feel ea NE: (4) 


We may ask when these are sufficient conditions. The 
best answer one can manage in a general case is that if 
all the second partial derivatives of D with respect to 
the z,’s and y;’s exist, then the critical point determined 
by conditions (3) and (4) is a minimum if the matrix 
whose 7th row and jth column element is 


a D 
Op: Op; critical eee 


where the p’s are the x’s and y’s, is positive definite. 
In a specific case, one may determine whether or not 
the matrix is positive definite or one may simply find 
all the critical points (7.e., those satisfying necessary 
conditions) and evaluate D at each. The absolute minimum 
must be at one of the critical points since ‘‘end points” 
can be easily ruled out. 
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The sort of f one would want to use would be a goo 
metric function, 7.e., f(x) is monotonically nondecreasin 


{(0) = 0 
f(z) = f(—2). 


If we require that f(x) be monotonically increasing (with a 
then (1) implies 


l| 


y; | j Saye 


which implies (since y;-, and y; should not coincide) tha 
ti = (Yi + Yi-1)/2 j= 2 


(x; is halfway between y; and y;-1). 
We now take a specific example of f(x) to furthe 
illuminate the situation. 
Let fa) =a" | 


eat, Ue eps |e 


(3) implies 


= (9; + Yi) /2 OF 9; = 25; Sea 
(p= 2, e aM, (G 
(4) implies 
[ @-wr@a@=0 j=1,--,N. | 


That is, y; is the centroid of the area of p(x) between 2 
and Birra. 

Because of the complicated functional relationship 
which are likely to be induced by p(x) in (6), this i 
not a set of simultaneous equations we can hope to solv 
with any ease. Note, however, that if we choose y, cor 
rectly we can generate the succeeding x;’s and y;’s b. 
(5) and (6), the latter being an implicit equation for 2;, 
in terms of x; and y;. 

A method of solving (5) and (6) is to pick y,, caleulat 
the succeeding x;’s and y,’s by (5) and (6) and then i 
yw 18 the centroid of the area of p(w) between vy and & 
yi was chosen correctly. (Of course, a different choice » 
appropriate to each value of NV.) If yy is not the appre 
priate centroid, then of course y, must be chosen agair 
This search may be systematized so that it can be pel 
formed on a computer in quite a short time.’ | 

This procedure has been carried out numerically on th 
IBM 709 for the distribution p(z) = 1/+/2z2 e*'”, unde 
the restriction that ¢y/24; = 0 for N even, and yvvs1)/2 = 
for N odd. This procedure gives symmetric results, 7.¢ 


‘ Obtaining explicit solutions to the quantizer problem for 
nontrivial p(x) is easily the most difficult part of the problen 
The problem may be solved analytically where p(x) = 1/\/2m e-z? 
only for N = 1,N = 2. ForN =1, m1 = —o,y =0,m = +0 
For N = 2, 21 Fe Ey eS —9/2/m, x2 = 0, Ya Vv 2/1, 3 = +e 
(1/2/m is the centroid of the portion of 1/+/2x e-z2/2 between tl 
origin and +.) For N = 3, some sort of numerical estimation 
required. A somewhat different approach, which yields resul 
somewhat short of the optimum, is to be found in V. A. Gamas 
“Quantization of signals with non-uniform steps,”’ Electrosvyaz, ve 
10, pp. 11-13; October, 1957. 
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‘a signal amplitude x is quantized as Y,, then —2 is 
lantized as —y,. The answers appear in Table I on 
uge 11. 

An attempt has been made to determine the functional 
pendence of the distortion on the number of output 
vels. A log-log plot of the distortion vs the number of 
ttput levels is in Fig. 1. The curve is not a straight line. 
ne tangent to the curve at N = 4 has the equation 
= 1.82 N~* and the tangent at N = 36 has the 
juation D = 2.21pq°"°*. One would expect this sort 
behavior for large V. When JN is large, the amplitude 
‘obability density does not vary appreciably from one 
id of a single input range to another, except for very 
rge amplitudes, which are sufficiently improbable so 
rat their influence is slight. Hence, most of the output 
vels are very near to being the means of the end points 
' the corresponding input ranges. Now, the best way 
' quantizing a uniformly distributed input signal is to 
race the output levels uniformly and to put the end 
ynts of the input ranges halfway between the output 
vels, as in Fig. 2, shown for N = 1. The best way of 
roducing a quantizer with 2N output levels for this 
istribution is to divide each input range in half and 
at the new output levels at the midpoints of these 
inges, as in Fig. 3. It is easy to see that the distortion 
the second case is } that in the first. Hence, D = kN~” 
here k is some constant. In fact, k& is the variance of 
re distribution. 

‘If this sort of equal division process is performed on 
ich input range of the optimum quantizer for a normally 
istributed signal with N output levels where N is large 
1en again a reduction in distortion by a factor of 4 is 
<pected. Asymptotically then, the equation for the 
yngent to the curve of distortion vs the number of output 
vels should be D = kN~’ where & is some constant. 
Commercial high-speed analog-to-digital conversion 
juipment is at. present limited to transforming equal 
iput ranges to outputs midway between the ends of 
1e input ranges. In many applications one would like 
» know the best interval length to use, 7.e., the one 
ielding minimum distortion for a given number of output 
vels, NV. This is an easier problem than the first, since 
is only two-dimensional (for N = 2), z.e., D is a func- 
on of the common length 7 of the intervals and of 
ay particular output level, y,. If the input has a sym- 
etric distribution and a symmetric answer is desired, 
1e problem becomes one dimensional. If p(x) is the input 
mplitude probability density and f(x) is the function 
ich that the distortion D is Elf(souz — Sin)], then, for 
n even number 2N of outputs, 


2%, [te [Aa boo 


foo} 
(x - 
(N-1)r 


sf 


Max: Quantizing for Minimum Distortion 9 


For a minimum we require 


h r(« = [+ bp dx = 0. (8) 


(N-1)r 


(N21) 


A similar expression exists for the case of an odd number 
of output levels. In either case the problem is quite 
susceptible to machine computation when f(x), p(x) and N 
are specified. Results have been obtained for f(z) = 2’, 
p(x) = 1//2n e* ”, N = 2 to 36. They are indicated 
in Table II on page 12. 

A log-log plot of distortion vs number of output levels 
appears in Fig. 1. This curve is not a straight line. The 
tangent to the curve at N = 36 has the equation D = 
1.47 N-***. A log-log plot of output level spacing vs 
number of outputs for the equal spacing which yields 
lowest distortion is shown in Fig. 4. This curve is also 
not a straight line. Lastly, a plot of the ratio of the distor- 
tion for the optimum quantizer to that for the optimum 
equally spaced level quantizer can be seen in Fig. 5. 


Key To THE TABLES 


The numbering system for the table of output levels, 
y;, and input interval end points, x;, for the minimum 
mean-squared error quantization scheme for inputs with 
a normal amplitude probability density with standard 
deviation unity and mean zero is as follows: 


For the number of output levels, NV, even, x, is the 
first end point of an input range to the right of the 
origin. An input between x; and 2;., produces an 
output y;. 

For the number of output levels, NV, odd, y, is the 
smallest non-negative output. An input between 2;_; 
and x; produces an output y;. 


This description, illustrated in Fig. 6, is sufficient because 
of the symmetry of the quantizer. The expected squared 
error of the quantization process and informational 
entropy of the output of the quantizer are also tabulated 
for the optimal quantizers calculated.” (If p, is the prob- 
ability of the kth output, then the informational entropy 
is defined as — >o*_, pz loge pi.) 

Table II also pertains to a normally distributed input 
with standard deviation equal to unity. The meaning of 
the entries is self-explanatory. 


2 The values of informational entropy given show the minimum 
average number of binary digits required to code the quantizer 
output. It can be seen from the tables that this number is always a 
rather large fraction of log, NV, and in most cases quite near 0.9 
log, NV. In the cases where V = 2”, n an integer, a simple n binary 
digit code for the outputs of the quantizer makes near optimum use 
of the digital transmission capacity of the system. 
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Fig. 1—Mean squared error vs number of outputs for optimum 
quantizer and optimum equally spaced level quantizer. (Minimum 
mean squared error for normally distributed input with « = 1.) 
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Fig. 2—Optimum quantization for the uniformly distributed case, 
N = 1. (Short strokes mark output levels and long strokes mark 


end points of corresponding input ranges.) 
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Fig. 3—Optimum quantization for the uniformly distributed case, 
N = 2, (Short strokes mark output levels and long strokes mark 
end points of corresponding input ranges. ) 
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Fig. 4—Output level spacing vs number of output levels for equa 
optimum case. (Minimum mean squared error for normall; 
distributed input with « = 1.) 
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Fig. 5—Ratio of error for optimum quantizer to error for optimur 
equally spaced level quantizer vs number of outputs. (Minimun 
mean squared error for normally distribured input with « = 1! 


N EVEN 


i L 1 


N ODD 


| 1 1 


Yin Mayo Nanya 


Fig. 6—Labeling of input range end points and output levels f: 
the optimum quantizer. (Short strokes mark output levels a 
long strokes mark input range end points.) 


TABLE I 
PARAMETERS FOR THE OPTIMUM QUANTIZER 


Xj Yi Xj Yi Xj Yi 
— 0.0 0.0 0.7980 | 0.0 1.224 
0.6120 
1.000 0.3634 0.1902 
0.0 1.000 1.536 
Uj Yi Uj Yi 5 Yi 
0.0 0.4528 | 0.3823 | 0.0 0.0 0.3177 
0.9816 | 1.510 1.244 0.7646 | 0.6589 | 1.000 
Lai 24: 1.447 1.894 
0.1175 0.07994 0.05798 
1.911 2.203 2.443 
Nii N= —8 N=9 
Uj Yi Xj Yi Xj Yi 
0.2803 | 0.0 0.0 OR2a aU OL22185 020 
0.8744 | 0.5606 | 0.5006 | 0.7560 | 0.6812 | 0.4436 
1.611 1.188 1.050 1.344 1.198 0.9188 
74 UBS 1.748 22152 1.866 1.476 
Dy PAD 
0.04400 0.03454 0.02785 
2.647 2.825 2.983 
Nel) Nae —aatal Nee k2 
Xj Yi Xj Yi Xj Yi 
0.0 0.1996 | 0.1837 | 0.0 0.0 0.1684 
0.4047 | 0.6099 | 0.5599 | 0.3675 | 0.3401 | 0.5119 
0.8339 | 1.058 0.9656 | 0.7524 | 0.6943 | 0.8768 
17325 1.591 1.4386 Lat9 1.081 1.286 
1.968 2.345 2.059 1.693 1.534 1.783 
2.426 2.141 2.499 
0.02293 0.01922 0.01634 
HH] ) 5 1745) 8) 188) 3.372 
_—————— 
N = 13 N = 14 N = 15 
vj Yi Xj Yi Xj Yi 
ao 0.1569 | 0.0 0.0 0.1457 | 0.1369 | 0.0 
2 0.4760 | 0.3138 | 0.2935 | 0.4413 | 0.4143 | 0.2739 
33 0.8126 | 0.6383 | 0.5959 | 0.7505 | 0.7030 | 0.5548 
4 1.184 0.9870 | 0.9181 | 1.086 1.013 0.8512 
5 1.623 1.381 1 P47FZE 1.468 lmao ila iis) 
6 Yes) 1.865 1.703 1.939 1.776 1.546 
7 2.565 2,282 2.625 2.344 2.007 
8 2.681 
itor 0.01406 0.01223 0.01073 
Entropy 3.481 3.582 3.677 
EEE 
Nelo INE 1 N = 18 
Li Yi Xj Yi Xj Yi 
ira 0.0 OFIZ84 120, 1215) 1-050 0.0 0.1148 
2 0.2582 | 0.3881 | 0.3670 | 0.2430 | 0.2306 | 0.3464 
3 0.5224 | 0.6568 | 0.6201 | 0.4909 | 0.4653 | 0.5843 
4 0.7996 | 0.9424 | 0.8875 | 0.7493 | 0.7091 | 0.8339 
5 1.099 1.256 WS: 1.026 0.9680 | 1.102 
6 1.437 1.618 1.508 ier jing PABIL 1.400 
ih 1.844 2.069 1.906 1.685 1 Syehe) 1.746 
8 2.401 PETE: 2.454 Oh APA 1.964 2.181 
2.781 2.504 2.826 
Error 0.009497 0.008463 0.007589 
Entropy 3.765 3.849 3.928 


NEO N = 20 N = 21 
Xj Yi 2; Yi Xj Yi 
PS! 0.1092 | 0.0 0.0 0.1038 | 0.09918 | 0.0 
2 0.3294 | 0.2184 | 0.2083 | 0.3128 | 0.2989 | 0.1984 
3 0.5551 | 0.4404 | 0.4197 | 0.5265 | 0.5027 | 0.3994 
+ 0.7908 | 0.6698 | 0.6375 | 0.7486 | 0.7137 | 0.6059 
5 1.042 | 0.9117 | 0.8661 | 0.9837 | 0.9361 | 0.8215 
6 1.318 ible} Walid 1.239 Te tlieAss 1.051 
a 1.634 1.464 1.381 1.524 | 1.440 1.300 
8 2.018 | 1.803 1.690 1.857 1.743 1.579 
9 2.550 | 2.232 | 2.068 | 2.279 | 2.116 1.908 
10 2.869 | 2.594 | 2.908 | 2.685 | 2.324 
11 2.946 
Error 0.006844 0.006203 0.005648 
Entropy 4.002 4.074 4.141 
INE = 2 Ne = 23 N = 24 
vj Yi vj Yi Xj Yi 
Z=1 0.0 0.09469 | 0.09085 | 0.0 0.0 0.08708 
2 0.1900 | 0.2852 | 0.2736 | 0.1817 | 0.1746 | 0.2621 
3 0.3822 | 0.4793 | 0.4594 | 0.3654 | 0.3510 | 0.4399 
+h 0.5794 | 0.6795 | 0.6507 | 0.5534 | 0.5312 | 0.6224 
5 0.7844 | 0.8893 | 0.8504 | 0.7481 | 0.7173 | 0.8122 
6 1.001 1.113 1.062 | 0.9527 | 0.9122 | 1.012 
7 1.235 1.357 1298 1,172 1.119 1.227 
8 1.495 1.632 1.546 1.411 1.344 1.462 
9 1.793 1.955 1.841 1.681 1.595 1.728 
10 2.160 | 2.366* || 2.203 |) 2.000 1.885 | 2.042 
11 2.674 | 2.982) | 2.711 2.406 | 2.243 | 2.444 
12 3.016 | 2.746 | 3.048 
Error 0.005165 0.004741 0.004367 
Entropy 4.206 4,268 4.327 us 
Nie=—25 N = 26 Nie 
Xj Yi Xj Yi Xj Yi 
Fe 0.08381 | 0.0 0.0 0.08060 | 0.07779 | 0.0 
2 0.2522 | 0.1676 | 0.1616 | 0.2425 | 0.2340 | 0.1556 
3 0.4231 | 0.3368 | 0.3245 | 0.4066 | 0.3921 | 0.3124 
4 0.5982 | 0.5093 | 0.4905 | 0.5743 | 0.5537 | 0.4719 
5 0.7797 | 0.6870 | 0.6610 | 0.7477 | 0.7202 | 0.6354 
6 0.9702 | 0.8723 | 0.8383 | 0.9289 | 0.8936 | 0.8049 
Fle i 8} 1.068 | 1.025 Heal 1.077 | 0.9824 
8 1.394 1.279 1224) 1 1.3828 e273 ise 
9 1.641 1.510 1.442 1.556 1.487 1.374 
10 1.927 ti 1.685 1.814 | 1.727 1.599 
11 2.281 2.083 1.968 | 2.121 | 2.006 1.854 
12 2.779 | 2.480 | 2.318 | 2.514 | 27385225158 
13 3.079 | 2.811 | 3.109 | 2.842. | 2.547 
14 3.137 
Error 0.004036 0.003741 0.003477 
Entropy 4.384 4.439 4.491 
a ee ee 
N= 28 N = 29 N = 30 
vj Yi uj Yi vj Yi 
j=l 0.0 0.07502 | 0.07257 | 0.0 0.0 0.07016 
2 0.1503 | 0.2256 | 0.2182 | 0.1451 | 0.1406 | 0.2110 
3 0.3018 | 0.3780 | 0.3655 | 0.2913 | 0.2821 | 0.3532 
4 0.4556 | 0.5333 | 0.5154 | 0.4896 | 0.4255 | 0.4978 
5 0.6132 | 0.6930 | 0.6693 | 0.5912 | 0.5719 | 0.6460 
6 0.7760 | 0.8589 | 0.8287 | 0.7475 | 0.7225 | 0.7990 
7 0.9460 | 1.033 | 0.9956 | 0.9100 | 0.8788 | 0.9586 
8 1.126 12S eal iz 1.081 1.043 1.127 
9 1.319 1.419 1.362 1.263 Ue 2aled 1.306 
10 1.529 1.640 1.570 1.461 1.404 1.501 
11 1.766 1.892 1.804 1.680 1.609 1 Bally 
12 2.042 | 2.193 | 2.077 1.929 1.840 1.964 
13 2.3855 2.078 \o2-417 28226" 20 2.258 
14 2.871 3.164 | 2.899 | 2.609 | 2.448 | 2.638 
15 3.190 | 2.926 | 3.215 
Error 0.003240 0.003027 0.002834 
Entropy 4.542 4.591 4.639 


Cont'd next page 
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Taste I, Cont'd TABLE II 
Ne=eoit N = 32 NB=EDS PARAMETERS FOR THE OPTIMUM EQuaLLy SpaceD LEVEL QUANTIZER 
; ; : , A Number Output | Output Level | Mean Squared | Informational 
vj Yi uj Yi vj Yi Levels P Soaoiie Error Entropy 
j=1 | 0.06802 | 0.0 0.0 0.06590 | 0.06400} 0.0 Gin été 
2 | 0.2045 | 0.1360 | 0.1320 | 0.1981 | 0.1924 | 0.1280 1 ae oe TORO 
3 | 0:3422 | 0.2729 | 0.2648 | 0.3314 | 0.3218 | 0.2567 2 1.596 eens aaa 
4 0.4822 | 0.4115 | 0.3991 | 0.4668 | 0.4530 | 0.3868 3 1.224 Cee ong 
5 0.6254 | 0.5528 | 0.5359 | 0.6050 | 0.5869 | 0.5192 4 0.9957 Oe ye 
6 | 0.7730 | 0.6979 | 0.6761 | 0.7473 | 0.7245 | 0.6547 5 0.8430 0. boone Saas 
7 0.9265 | 0.8481 | 0.8210 | 0.8947] 0.8667 | 0.7943 6 0.7334 0. ee 
8 1.088 | 1.005 | 0.9718 | 1.049 | 1.015 | 0.9392 7 0.6508 UB te po 
9 P50) ete 1180 15200 oi Terra Gor 8 0.5860 0.0374 oad 
10 daar V1 847 >. 2909.°| 1.387 | 1,338 4) 1.252 9 0.5338 0.03069 aes 
11 15646. | e540. 482° 101.677 1518 Lik ae 10 0.4908 0.02568 as 
12 W875, F 17581682) 1,788 el 76. ot O12: 11 0.4546 0.02185 Hee 
13 2.143 | 1.997 | 1.908 | 2.029 | 1.940 | 1.821 12 0.4238 0.01885 ae 
14 ATT “Wc2. 980 12-174 19.319 1.2904" | 9.060 13 0.3972 0.01645 3. ee 
15 | 2.952 | 2.665 | 2.505 | 2.692 | 2.538 | 2.347 14 0.3739 0.01450 a 
16 3.239 Sc 977 | S2265. | Bx002 “eo. 718 15 0.3534 0.01289 3.524 
17 3.285 16 0.3352 0.01154 3.602 
17 0.3189 oe 3.676 
18 0.3042 0.0094 
Error 0.002658 0.002499 0.002354 ' Mb er ae ot 3 ; i 
20 0.2788 0.00786 
nage ae as ge: 21 0.2678 0.007235 3.933 
22 0.2576 0.006678 3.990 
N = 34 N = 35 N = 36 23 0.2482 0.006185 4.045 
24 0.2396 0.005747 4.097 
xj Yi xj Vi xj Yj 25 0.2315 0.005355 4.146 
26 0.2240 0.005004 4.194 
j=1 0.0 0.06212 | 0.06043 | 0.0 0.0 0.05876 27 0.2171 0.004687 4.241 
2 | 0.1244 | 0.1867 | 0.1816 | 0.1209 | 0.1177 | 0.1765 28 0.2105 0.004401 4.285 
3 0.2495 | 0.3122 | 0.3036 | 0.2423 | 0.2359 | 0.2952 29 0.2044 0.004141 4.328 
4 | 0.3758 | 0.4394 | 0.4272 | 0.3650 | 0.3552 | 0.4152 30 0. 1987 0.003905 4.370 
5 | 0.5043 | 0.5691 | 0.5530 | 0.4895 | 0.4762 | 0.5372 31 0.1932 0.003688 4.410 
6 | 0.6355 | 0.7020 | 0.6819 | 0.6166 | 0.5996 | 0.6620 32 0.1881 0.003490 4.449 
7 | 0.7705 | 0.8391 | 0.8146 | 0.7471 | 0.7261 | 0.7903 33 0.1833 0.003308 4.487 
8 | 0.9104 | 0.9818 | 0.9523 | 0.8820 | 0.8567 | 0.9231 34 0.1787 0.003141 4.524 
9 1.057 | 1.181 | 1.096 | 1.023 | 0.9923 | ¥.062 35 0.1744 0.002986 4.560 
10 12210, 15290. 9 248 1.170) 12134. «| 207. 36 0.1703 0.002843 4.594 
11 1.375 | 1.460) | 1.411 | 1.327 | 1.285. | 1.362 
12 1.553 | 1.646 | 1.587 | 1.495 | 1.445 | 1.528 
13 1.749 | 1.853 | 1.781 | 1.679 | 1.619 | 1.710 
14 Oven 2090) Wo2s00% 4.4883 \ch 8128 10013 
15 192. 23201) 2.376 | 2.260 | 2.119... | 2,030 | 9. 146 
16 | 2.559 | 2.743 | 2.584 | 2.401 | 2.287 | 2.427 
17 | 3.025 | 3.307 | 3.048 | 2.767 | 2.609 | 2.791 
18 3.328 | 3.070 | 3.349 
Error 0.002220 0.002097 0.001985 
Entropy 4.815 4.856 4.895 
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A Note on P-nary Adjacent-Error-Correcting Codes* 


Summary—Binary group codes described by Abramson permit 
e correction of all single errors and all double errors in adjacent 
gits,. with the use of significantly fewer check digits than codes 
pable of correcting all double-bit errors. 

'This note considers the generalization of Abramson’s codes to 
@ p-nary case, where a symbol alphabet consisting of the digits 
1, ---, — 1 is used for transmission, p being a prime number. 
xamples of such p-nary codes are given, as well as necessary 
mditions for their existence. These codes bear the same relation 
the p-nary Golay codes as Abramson’s codes do to the familiar 
amming codes. 

Some as yet unanswered questions are raised, and suggestions 
r further possible generalizations are given. 


I. INTRODUCTION 


| ECENT work by Abramson’ has described the con- 
struction of binary group codes which are rather 
efficient for combatting burst noise. These single- 
ror-correcting double-adjacent-error-correcting codes 
SEC-DAEC codes) use significantly fewer check digits 
ian codes capable of correcting all double-bit errors. 
urthermore, the digital circuitry” required for imple- 
enting these codes is extremely simple. 
This note describes a generalization of binary SEC- 
»AEC codes to the p-nary case, where a symbol alphabet 
onsisting of the digits 0, 1, --- , p — 1 is used for trans- 
ission, p being a prime number. These codes bear the 
me relation to the p-nary Golay codes’ as Abramson’s 
odes do to the Hamming SEC codes. 
‘In the binary case, the block length n is related to 
1e number of check digits k by the relation, n = 2" — 1. 
his relation derives from the requirement that the 
umber 2” of corrector combinations possible with k 
wity checks be at least 1 + 2n in order to cover all n 
ngle-bit errors and all n double-adjacent-bit errors, as 
ell as the case of no errors per block. Note that an error 
_ the first and last digits of a block is counted as a 
vorrectible) double-adjacent error. Observe also that 
cactly one k-bit combination remains unused for correc- 
on. 


* Manuscript received by the PGIT, September 21, 1959. 
+ Stanford Research Institute, Menlo Park, Calif. 
-1N. Abramson, “A Class of Systematic Codes for Non-Inde- 
mdent Errors,’’ Stanford Univ. Electronics Lab., Stanford, Calif., 
ept. No. 51; December 30, 1958. Also: IRE Trans. on Inror- 
ation TuEory, vol. IT-5, pp. 150-157; December, 1959. 
-2N,. Abramson and B. Elspas, “Double-Error-Correcting En- 
ders and Decoders for Non-Independent Binary Errors,’’ pre- 
nted at the Internatl. Conf. on Information Processing, Paris, 
rance; June 15-22, 1959. < 
3M. J. E. Golay, “Notes on digital coding,’ Proc. IRH, vol. 37, 
657; June, 1949. 


BERNARD ELSPASt 


Il. THe P-nary Case 


In the p-nary generalization, the digit in any particular 
position may be in error in p — 1 ways, and any pair 
of digit errors (in given locations) may occur in (p — 1)? 
ways. Hence for a p-nary SEC-DAEC code we require that 


1 Pip 1) np Dae ai = Des pit) 


The left-hand side of the above inequality leaves a 
remainder of 1 upon division by p, while the right-hand 
side is divisible by p. It follows that integer solutions 
for n and k are constrained by 


a sili be 


a (2) 
It is shown below that (at least for some values of k and p) 
SEC-DAEC codes exist for which the equality sign holds 
in the above condition. This relation between the block 
length and the number of check digits reduces to the 
binary formula when p = 2. 

Any group code* is completely specified by a listing 
of the parity checks (modulo-p) which must be satisfied 
by a valid code word. Alternatively, this listing may 
be viewed as a specification of the n characteristics (2.e., 
the correctors which result at the receiver when a single 
digit is received in error by an amount +1). Each charac- 
teristic, A;, is a k-digit p-nary number. The corrector 
for an error of amount e occurring in the zth digit is then, 
by linearity of the parity checks, equal to eA;. Likewise, 
the double-error corrector corresponding to an error of 
amount e in the 7th digit and an error of amount f in 
the jth digit occurring simultaneously is then eA, + fA,. 
Thus if all single errors and all double-adjacent errors 
are to be correctible, the np(p — 1) correctors, eA; + 
fA sax (where e me ele ie Lf =O es seared 
2 = 1, 2, --- , n and all subscripts are taken modulo n), 
must all be distinct. Note that of the p” conceivable 
correctors, p — 1 remain unused for correction. The 
all-zeros corrector is, of course, reserved for the indication, 
“no error.” 


III. Exampues or P-NARY SEC-DAEC Copss 


The simplest example of such a code is furnished by 
the ternary (p = 3) case for k = 4, n = 13. Here the 
thirteen required characteristics, A;, are as shown in 
Fig. 1. The corresponding parity-check equations, in 
Fig. 2, are obtained by turning the list of characteristics 


4D. Slepian, ‘A class of binary signalling alphabets,’’ Bell Sys. 
Tech. J., vol. 35, pp. 203-234; January, 1956. 
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Unit Error in 


Digit Number Corrector 
1 1001 
2 0101 
3 1011 
4 1101 
5 aba 
6 atslal 
2211 
8 0221 
9 1021 

10 2101 
11 1211 
12 0121 
13 OO11 


Fig. 1—Correctors for ternary SEC-DAEC code with n = 13 and 


i = 


Data Digits-————————— _ >———Checck Digits 
Sg bs He 2ate: Day a he ea ee ee 
ae + Qa, + 2rg + + th + 2yo => Ys 
Y2 + 2Y3 + Y4 
Ys + Ys 


foe ae hy ap 
2 apes cP ORS yp ae Pins Se ZN aR Se 
two vam Apap yap eta Uh ae Y2 + 


Fig. 2—Parity check relations for the ternary SEC-DAEC code with n = 13 and k 


1O11122012100,... 


Fig. 3—Ternary feedback shift register with cycle length of 13. 


on its side, yielding four parity checks over the thirteen 
code digits. One of these parity checks is simply a mod-3 
sum of all the code digits, while the other three parity 
checks follow a particular pattern, cyclically permuted. 
Consideration of the A,’s shows that all of the 78 four- 
digit ternary numbers represented by A;, 24;, A; + Aisi, 
2A; + 2Aii:, A: + 2Aj41, 2A; + Ais: are distinct and 
nonzero, thus permitting the correction of the 26 pos- 
sible single errors and the 52 possible double-adjacent 
errors. 

A similar code (also ternary) has been found for k = 6, 
nm = 121, which corrects all 242 possible single errors 
and all 484 possible double-adjacent errors in a block 
of 121 ternary digits by using six parity checks. 

Both of the above codes were constructed by finding 
suitable feedback shift-register sequences to determine the 
cyclic parity check patterns. These shift-register sequences 
satisfy recursion relations of degree k — 1 (corresponding 


to shift registers with k — 1 stages), and their period 
(cycle length) is n = (3° " — 1)/2. For the code with 
k = 4, the recursion relation is 

Gs cg By, at ky (mod 3), 


corresponding to the characteristic polynomial, 


+2 +2°= 0, 


which is irreducible over the ternary field (0, 1, 2), ax 
which has period 13. The feedback shift-register employiz 
this recursion relation is shown in Fig. 3. 

The code for k = 6 can be constructed similarly - 
terms of the recursion relation, 


A a et oF (mod 8) 


Ui+4 
which is equivalent to the characteristic polynomial, 
z+ 2¢+2=0, 
of period 121. 


IV. Discussion 


There is an extensive theory of feedback shift-registe 
and their characteristic polynomials; see for example t 
paper by Elspas.° This theory shows that the desire 
cycle length, », as given by (2), can always be realize 
for arbitrary k and p with several different shift-regist 
connections. However, it can be shown (see Appendi 
that two additional conditions must be met in order f 
these sequences to be applicable to the construction 


> B. Elspas, ‘The theory of autonomous linear sequent 
networks,’’? IRE Trans. on Crrcvir THEory, vol. CT-6, p. 45- 
March, 1959. 
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1C-DAKC codes. The simpler of these conditions re- 
ires that n be relatively prime to p — 1. This condi- 
m, has the effect of ruling out, e.g., all even values of n 
e., odd values of k) for p (a prime) greater than 2. 
The other condition requires that if the characteristic 
lynomial, f(x), divides (without remainder) a poly- 
mial of the form, ax**' + be* — ex — d, where a + 
=c + d, then a = ¢, and b = d. It is not presently 
nown for which values of p and k it is possible to find 
least one polynomial of degree k — 1 and period n 
at satisfies this condition, other than the two examples 
ed. However, it seems not unreasonable to suspect that 
ner cases exist. 

The situation for p = 2 (the binary SEC-DAEC codes 
scribed by Abramson’) is much more clear cut since 
e two extra conditions are then automatically satisfied 
r all characteristic polynomials having the proper period, 
“= 2°" — 1. Thus Abramson’s codes exist (in shift- 
mister form) for all values of k > 4. When k = 1, 2, 
3, the situation is degenerate, since then n < k, and 
+ information can be transmitted. 

The broader problem of the existence of SEC-DAEC 
des not derivable from shift-register sequences, is still 
. open question (although one such case’ is known for 
= 2, = 15). A possible generalization to nonprime 
oduli (along the lines given by Cocke’) might also 
wrant investigation. Another possibly interesting gener- 
zation concerns the extension to p-nary codes of the 
wrk of Fire’ on binary burst-error codes where the 
irst length is greater than two. 


V. APPENDIX 


NEcESSARY CONDITIONS FOR THE EXISTENCE OF 
P-nary SEC-DAEC Cops 


Theorem: In order for a p-nary shift-register having 
e (irreducible) characteristic polynomial f(x) to be appli- 
ble to the construction of a p-nary SEC-DAEC code 
n digits with k check digits, the following conditions 
ust be satisfied: 


1) Register cycle length = n = (p** — 1)/(p — 1) must 
be relatively prime to p — 1. 


8 J. Cocke, “Lossless symbol coding with nonprimes,’’ TRANs. 
{E on INFORMATION Tueory, vol. IT-5, pp. 33-34; March, 1959. 
7 P. Fire, “A Class of Multiple-Error-Correcting Binary Codes 
¢ Non-Independent Errors,” Sylvania Reconnaissance Sys. Lab., 
fountain View, Calif., Rept. No. RSL-E-2; March, 1959. 
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2) If f(x) divides a polynomial of the form, 
ax’** + ba" — cx — d 


where a + 6b = c+d, thena = 
hold. 


Proof. It follows directly from (2), and the discussion 
of Section III, that the register cycle length must be 
given by 


cand 6 = d must 


Thus f(x) divides 2” — 1. 
1) If n is not relatively prime to p — 1, then 
n=td and p—1= 4d 

for some integers, t and g. We now show that this implies 
that the resulting code based on f(«) would leave an error 
of a certain amount in digit 1 indistinguishable from a 
unit error in digit t + 1. Thus this code would not even 
be single-error-correcting. We have: 


t 


go —t=ax*-1=y°—1, where y=a. 


Now since d divides p — 1, y* — 1 may be factored® 
into linear factors, 


d 
y—-1=[]y-a) (modp), 
i=1 
where the a; are integers in the set (1, 2, --- , p — 1). 
Since f(x) divides x" — 1, and is irreducible (mod p), 
it follows that f(x) divides some factor x’ — a,. But 


this in turn means that the shift-register sequence repeats 
(except for a constant factor a;) in blocks of length ¢ 
(a divisor of n). Such a sequence cannot be used for 
single-error correction of n digits since it leaves an error 
of amount a; in digit 1 indistinguishable from a unit 
error in digit ¢ + 1, for example. Hence n must be prime 
hKODY ONT 

2) If f(x) divides x*(ax + b) — (cx + d) witha + b = 
c + d, then, as a little thought will show, errors of amounts 
d and ¢, respectively, in digits 1 and 2 will produce the 
same pattern of parity check violations as errors of 
amounts b and a, respectively, in digits h + l andh + 2. 
This means that the code is not adjacent-error-correcting, 
unless h is a multiple of n, and consequently, a = ¢ 
and b = d. 


8 See, for example, L. E. Dickson, “Introduction to the Theory 
of Numbers,’’ Dover Publications, Inc., New York, N. Y., Theorem 
20, p. 1621957. 
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+S, * 
Codes for the Correction of “Clustered” Errors 
SIEGFRIED H. REIGERT 


Summary—A method is described which permits the systematic 
construction of codes capable of error-free transmission, provided 
errors occur in ‘‘clusters’’ of limited duration. The method is valid 
for error clusters of any prescribed duration. The codes are rela- 
tively easy to implement and decoding operations are straight- 
forward. Specific examples are given and applications to teletype 
transmission are discussed. 


I. INTRODUCTION 


HE construction of most error-correcting codes, 
Slices have been described in the literature, is 

based on the assumption that errors occur inde- 
pendently in digit positions. Representative codes in this 
category were described by Hamming,’ Reed,” Slepian,” 
and Green and San Soucie.* With a digit error prob- 
ability p = 1 — gq, the usual assumption is that all pos- 
sible s-tuple errors in a block of n digits are equally 


likely. (There are exactly (") s-tuple errors with total 


probability ("\ptar*) 

In many instances, errors do not occur independently. 
On telephone lines, for example, impulse noise is prevalent 
and high error probability is confined to the duration of 
the impulse, the error probability being negligible at 
other times. Similarly, on certain types of fading radio 
circuits, the signal is buried in noise for short periods of 
time. Ordinary “‘static,’”’ caused by thunderstorms, fre- 
quently displays such burst-like characteristics. Abramson” 
has recently described a class of codes which are useful 
in such cases. His codes correct all single and adjacent 
double errors and require significantly less checking 
digits than codes capable of correcting all double errors. 

This paper describes a method which permits the con- 
struction of codes capable of correcting clustered errors 
of any prescribed cluster duration. The construction of 
these codes is simple and the decoding procedure is 
straightforward. 


II. GENERAL PROPERTIES OF SYSTEMATIC CODES 


This section introduces briefly some properties of sys- 
tematic codes which are significant in the subsequent code 
construction. The notation is identical to Slepian’s.* 


* Manuscript received by the PGIT, August 7, 1959. 

{| The RAND Corporation, Santa Monica, Calif. 

1R. W. Hamming, “Error detecting and error correcting codes,” 
Bell Sys. Tech. J., vol. 29, pp. 147-160; April, 1950. 

21. S. Reed, “A class of multiple-error-correcting codes and the 
decoding scheme,” IRE Trans. on INroRMATION THEORY, vol. IT-4, 
pp. 38-49; September, 1954. 

§ D. Slepian, “A class of binary signaling alphabets,’’ Bell Sys. 
Tech. J., vol. 35, pp. 204-234; January, 1956. 

4 J. H. Green, Jr., and R. L. San Soucie, “An error-correcting 
encoder and decoder of high efficiency,” Proc. IRE, vol. 46, pp. 
1741-1744; October, 1958. 

5.N..M. Abramson, ‘‘A Class of Systematic Codes for Non- 
Independent-Errors,” IRE Trans. oN INFoRMATION THEORY, vol. 
IT-5, pp. 150-157; December, 1959. 


In a systematic code, digit positions in any code wor 
of length n fall into two categories: information position: 
k in number, and m = n — & check positions. Entri¢ 
in the check positions are linear (modulo 2) combine 
tions of information digits and are usually called parit 
checks. Without loss of generality, we may let the info1 
mation digits occupy the first & places of each code worc 
Entries in the last (1 — k) positions are uniquely dete 
mined by parity checks over the first k digits. The parit 
check rules may be written 


k 
a; = ova; 4=kh+1,--: 5% ¢ 
7=1 


which is Slepian’s (9). In (1), a, is the digit in position - 
and is 0 or 1. Additions are modulo 2, vzz., 1 + 1 = 0 - 
0=0,0+1= 14 0 = 1. Eg: (1) defines m = = 
linear relations, one for each check digit. 

The key prob!em in the design of a code having specifie 
optimal properties is the determination of the matrix 7; 
No straightforward, general rules have as yet been foun 
to accomplish this. 

We modify (1) slightly for reasons which will becom 
apparent later. Each check digit will be allowed to perforr 
a parity check not only over information digits bu 
also over check digits in preceding places. Thus, th 
second check digit, a,.2., may also check a,,,, ete. Th 
check rules now read 

ti 


a; = C= 7s ( 


Qj; 4; 
which is similar to (1) except that the summation no: 
goes from j = 1 toz — 1. With a;; = 1, we may writ 
t=k+1,++- 4m. (2: 
In encoding, each check digit can still be written ag 
linear combination of information digits only, becaus 
d+: depends only on information digits and can | 
substituted by a linear combination of information digit 
in the equation for a,42, and so on. The code remair 
systematic as defined above. The distinction between (1 
and (2) lies in the decoding process. 

Let 6; be the digits after reception, 7 = 1, 2 --- % 
If an error occurred in position 7, we have b; ¥ ay, « 
a; + b; = 1 because of the rules for modulo 2 additio1 
If the digit in position 7 was received correctly, b; = 
or a; + b, = 0. It is possible, then, to introduce an 4 
place binary number, which we might call error vecto 
the 7th component of which is e; = a; + b; = 1, if € 
error occurred in position 7 and e,; = 0 otherwise. T} 
number of ones in the error vector is identical to t 


960 


umber of transmission errors. Also, any combination of 
rrors is defined uniquely by an error vector. 

As a first step in decoding, the decoder performs 
1 = n — k summations (modulo 2) as prescribed by 
he right-hand side of (2a); in other words, m digits are 
omputed as follows: 

] 


Da=s) nas; t= 1, oe 3 (3) 
g=1 


a;; = 0 for jee 
qg=i—k=1,2,---m. 


e may interpret the m values of r, as digits of an m-place 
inary number #, which we call Check Number. (Slepian 
lls it parity check sequence.) The modulo 2 sum of 
2a) and (3) is 


ratios »S; a:j(a; + b;) = Sane (4) 


7=1 

he check number for error-free transmission is the m- 
lace zero sequence. In addition to the all-zero sequence, 
) may possess any one of (2” — 1) values. Hence, a code 
wn correct a maximum of (2” — 1) distinct error combi- 
tions, which is possible only if an a;; matrix can be 
yund such that the (2” — 1) distinct error vectors (for 
‘ror combinations to be corrected) when introduced into 
t), generate (2” — 1) distinct check numbers. In this 
se the code is called ‘‘complete.”’ In general, codes are 
complete, because check numbers are interrelated and 
ust satisfy additional conditions. 

‘Consider the occurrence of a single error in position ¢ 
id designate the associated check number by A#,(1). 
nce a single error may occur in any one of 7 positions, 
1ere are n check numbers #,(1). The first digit of F,(1) 
1, = Qx+1,1, Which follows from (4) with e, = 1, and 
= 0, 7 + t. If the first check digit checks position 1, 
le have az1,. = 1, because of (2), and &,(1) has a 1 
the first position. In general, if R,(1) has a 1 in position 
‘parity check qg checks position ¢ and a... = 1. Thus, 
(1) is column ¢ of the a,;; matrix. We will subsequently 
fer to this fact as Rule I. Because of this one-to-one 
rrespondence between the a,; matrix and the array 
single-error check numbers F,(1), a systematic code 
y be described completely by either one.* The check 
imber concept is more useful in case of clustered errors. 
‘We now turn to relations between check numbers. 
ere are two reasons for such relations; first, the linear 
em of (2), (3) and (4), and second, the systematic 
ture of codes. An error vector, as defined above, iden- 
ying simultaneous errors in several positions is the 
nary sum of error vectors belonging to single errors 
those positions. Because of the linear form of (4), 
is property also holds for check numbers. Therefore, 
> check number for any multiple error is the linear (modulo 


6G. E. Sachs, ‘Multiple error correction by means of parity 
ecks,’? IRE Trans. on InrormatTion TueEory, vol. IT-4, pp. 
5-147; December, 1958. 
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2) sum of check numbers R,(1) for single errors in the error 
positions of the multiple error. We shall refer to this prop- 
erty as Rule II. 

The multitude of linear relations between check numbers 
is the reason why codes are, in general, incomplete. 
Additional restrictions are placed on the choice of check 
numbers by the nature of systematic codes. Imagine the 
portion of the R,(1) array identifying single errors in 
the last m positions of the code word. Because of a;; = 1, 
the first of these numbers must have a 1 in the first 
position, the second in the second position, etc. Further- 
more, since the first parity check may only include 
preceding positions, all numbers in this partial array 
except the first must have zeros in the first position, 
etc., because of Rule I. For example, in case of 5 check 
digits, the structure of the last five numbers in the array 
R.(1) must be as follows: 


oy (he eS 
Res (1) Leste aes 
Rees (1) (iis I ous 
Riss (1) PST GL aS (5) 
jie) e030. anes 
Ris (1) O05 O03 nl 


Entries in the starred places are still at our disposal as 
well as the entire check numbers belonging to the first & 
positions of the code word, subject to the restrictions 
imposed by Rule II and the condition that check numbers 
for correctable error combinations must all be different 
from each other. 


IIL. Coreck NUMBERS FOR CLUSTERED ERRORS 


The discussion in the previous section is quite general 
and applies to any systematic code governed by check 
rules as in (2). Consider now situations in which errors 
occur in clusters. The characteristic feature is the occur- 
rence of a high error probability for the duration of s 
successive digit positions. The error probability is negli- 
gible in the remaining (n — s) positions. Let us designate 
codes which are capable of effecting error-free transmission 
under these conditions as (n, k), codes. 

Up to length s, any combination of errors and correct 
digits is likely. Let h be the number of digit positions 
bracketed by the first and last error of an error combina- 
tion, these two positions included. There are 2”~” distinct 
error patterns of length h because errors may or may 
not occur in (h — 2) places between the first and last 
error position. Error patterns of length h may be iden- 
tified by h-place binary numbers (1 1) with a 1 
denoting an error and a 0 no error. For example, if h = 4, 
there are 2". = 2” = 4 distinct error patterns, in this 
notation (1001), (1101), (1011) and (1111). The total 
number of multiple error patterns of an (n, k), code is 
obtained by summation over h from 2 to s: 


dy Diet oe Sea (6) 
=2 
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Associated with these patterns are check numbers, 
which we write R,(1 --+ 1), where the subscript ¢ iden- 
tifies the position of the last error. For example, in case 
s =4, we obtain 2° — 1 = 7 check numbers with last 
error in position ¢, namely F#,(11), R.(101), #,(111), 
R,(1001), R,(1011), R,(1101), R,(1111). As a consequence 
of Rule II, all of these are linear combinations of F#,(1), 
Rea), Rioo(1) and R,-3(1). 

As the position of the last error is shifted through the 
code word, 2° * — 1 distinct sequences of multiple error 
check numbers occur, one for each error pattern. Hach 
of these sequences contains (n + 1 — h) members, 
because (n + 1 — h) is the number of distinct locations 
of an error pattern of length h in a code word of length n, 
2<h < s. In addition, there is the sequence of n single 
error check numbers R,(1) plus the all zero m-place 
number for correct transmission. Therefore, the total 
number of distinct check numbers required for an (n, k), 
code is 


No) =1+n+ See be hoy 


= (n + 2 — s)2°”’. (7) 


This defines a lower bound for m, the necessary number 
of redundant digits: 


a7 >(n+2— 82°” (8) 


where the equality would hold for a ‘‘complete’’ code. 
These relations have also been obtained by Fire.’ 

A further restriction on m arises as follows: suppose 
an error cluster of length s happens somewhere within 
the last m positions. The code has to cope with 2° possible 
errors vectors, which must be uniquely identifiable by 
the remaining (m — s) redundant digits. Therefore 


m > 2s. (9) 


For example, regardless of word length n, no (n, k), code 
(cluster length 5) can be found which has less than 10 
(redundant) check digits. 

A simple consideration shows that half of the error 
combinations contain an odd number of errors, if zero- 
error is included in the count for even numbers of errors. 


IV. GENERATION oF CoEecK NUMBERS 
WITH LINEAR SHIFT REGISTERS 


It has been demonstrated that (n, k)° codes are com- 


pletely defined by a set of n single error check numbers 
R,(1). Furthermore, several conditions have been dis- 
cussed, which must be met by the check numbers. As 
usual, there is no hint as to how to select an optimum 
set of check numbers. In principle, one could select the 
Jast m numbers in accordance with array (5) adjusting 


ai P. Fire, “A Class of Multiple-Error Correcting Binary Codes 
for Non-Independent-Errors,’”’ Stanford Electronics Labs., Stanford 
University, Stanford Calif., Tech. Rept. No. 55, Contract Nonr 
225(24), NR 373 360; April, 1959. 
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entries in the starred positions such that conflicts ¢ 
multiple error check numbers do not occur. One migk 
continue to select the remaining check numbers, on 
at a time, rechecking the entire array after each ste} 
Clearly, such a procedure would result in chaos for larg 
n and s. In addition, entirely untractable decoding rule 
would be the inevitable consequence. 

Linear feedback shift-registers offer a systematic metho 
of generating sequences which may be quite easily adapte 
for use as check numbers. We shall see that codes cor 
structed by this method are quite efficient, though gener 
ally incomplete. Moreover, the decoding process is straight 
forward and simple. 

A linear feedback shift-register is shown in Fig. 


Fig. 1—Linear feedback shift-register. 


(Their properties have been studied primarily by Zierle 
and Golomb.’) An m-stage shift register consists of 1 
binary memory devices (flip-flops F,, --- F,,), m switche 
Co, C1, *** Cm, Which are either open or closed, and (m — 1 
binary adders. Imagine the device driven by clock pulse: 
each of which shifts the flip-flop contents one positio 
to the right. Simultaneously, an input 2; is entered int 
the first stage where z; is given by 


m ™ 
Ca Sacra or 0 = SCs 
g=1 7=0 


and where ¢ =c, = 1. 
The following properties of sequences 2; are of sign 
ficance here: 


(i 


1) Sequences are periodic. Certain sequence generato: 
Ci ee Cm) generate sequences of peric 
(2” — 1), called maximum-length sequences. Max 
mum-length sequence generators have been tab 
lated up to m = 19 by Marsh” and are also liste 
up to length 9 in Albert.’* It is convenient to chara 
terize generators in octal equivalents of three-pla: 
binary numbers, vzz., 0 = 000, 1 = 001, 2 = GR 
3-= O11, 4 = 100) 6 =101,.6 31107 ee 
For example, the generator 1423 — (00110001001. 
identifies a 9-stage shift register in which the su 


8 N. Zierler, “Linear recurring sequences,” J. Soc. Indus. aw 
Appl. Math., vol. 7, pp. 31-49; March, 1959. 

*S. W. Golomb, “Sequences with Randomness Properties 
Glenn L. Martin Co., Baltimore, Md., Final Rept. on Contract N 
W36-039 Sc-54-3611; June 14, 1955. 

10 R. W. Marsh, “Table of Irreducible Polynomials Over GF ( 
Through Degree 19,” Natl. Security Agency, Washington, D. ¢ 
October, 1957. ; | 

uA. A. Albert, “Fundamental Concepts of Higher Algebr: 
University of Chicago Press, Chicago, IIl.; 1957. 
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. of the contents of stages 1, 5, 8, and 9 is fed back 
_ to the input. We shall call a maximum-length 
sequence an m sequence. 

There are exactly ¢(2” — 1)/m distinct m sequences 
, of length 2" — 1. 6(2” — 1) is the Euler function, 
, equal to the number of integers (1 included) smaller 
than (2” — 1) and coprime (no common factor 
except 1) with (2” — 1). 

3) The reverse of each m sequence has also period 
_ (2™ — 1). The corresponding pair of sequence 
generators C™ and C is related by c*¥ = ¢,,_;. 

4) An m sequence contains one run of m consecutive 
ones and one run of (m — 1) zeros. Also a run of 
(m — 2) ones, as well as a run of (m — 2) zeros 
occurs exactly once. In general, there are 2° runs 
of ones and 2’ runs of zeros of length (m — i — 2), 
for0 <7 < m— 8. Arun of m zeros does not occur. 
5) Consider the digits in each m consecutive positions 
as an m-place binary number. The result is a se- 
quence of (2” — 1) m-place binary numbers. Each 
occurs exactly once, with the exception of the m- 
place all-zero number. We will call this sequence 
the M sequence. 


In addition to the properties listed above, m sequences 
ssess the so-called “cycle and add” property which, 
fact, is their outstanding feature for our purposes. 
ne designation cycle and add means that a new sequence 
tained by adding termwise (modulo 2) an m sequence 
a cyclic permutation of itself is yet another cyclic 
‘rmutation of the original sequence. For example, if 
add consecutive digit pairs, we obtain the original 
sequence shifted by a certain number of positions. 
peating the process, we see that the property holds 
the sum of any digit pattern. Still more important, 
e cycle and add property also holds for the JZ sequence 
mply because it holds for each individual position in 
ie m-place numbers of the / sequence. 
Recal] now Rule II concerning check numbers. Suppose 
» take as single error check numbers, #,(1), a run of n 
nsecutive numbers from an M sequence. The cycle and 
d property then insures that the set of (n + 1 — h) 
eck numbers for a given error pattern of length h is 
run of (n + 1 — h) consecutive numbers in the M 
quence, and this is true for all error patterns, because 
altiple error check numbers are linear sums of single 
-or check numbers (Rule II). These runs of consecutive 
mbers are shifted with respect to each other in the 
sequence. Unfortunately, there is no simple explicit 
mula giving the amount of the various shifts as a 
netion of the sequence generator C’. The realizable code 
ord length n, given a cluster length s, and a number of 
eck digits m, depends only on C. The limitation is 
t no two check numbers may be identical. Identity 
check numbers would occur if any two runs of check 
mbers overlap in the M sequence. In order to deter- 
ne the maximum code length, it is only necessary to 
d the position of the first number in each run, and 


it is quite immaterial where in the 17 sequence we place 
the run of R,(1). A more detailed discussion of this 
procedure is found in the Appendix. 

The actual check numbers for a realizable code can 
be chosen in one of two ways. In the first method, the 
m-place check numbers are taken from an M sequence. 
Because of array (5) we have to place the run of R,(1) 
within the MW sequence so as to make F#,(1) that number 
which has zeros in the first (m — 1) places and a one 
in the last place. Thus, all check numbers are fixed and 
the check rules are found with Rule I. — 

It becomes clear now, why (2) had to be used instead 
of (1) to define the parity checks. Eq. (1) would require 
that all starred positions in array (5) be zeros; thus, 
a string of (m — 1) zeros would be needed in F,,(1) and 
Ry+1(1). There is only one run of (m — 1) zeros in an 
m sequence. Hence, parity check rules as in (1) are in- 
compatible with the properties of m sequences. 

An alternate method of check number assignment is 
based on the previously stated fact that half the error 
combinations contain an even number of errors (zero 
errors included), the other half containing an odd number 
of errors. This suggests a separation of these two groups 
by an over-all parity check to be performed by the last 
check digit in position n. Check numbers for even errors, 
then, have a zero in the last place and check numbers 
for odd errors have a one in the last place. The remaining 
(m — 1) positions (truncated check numbers) may now 
be taken from (JZ — 1) sequences. A slight complication 
arises, because an (JZ — 1) sequence does not contain 
the (m — 1)-place all-zero number. But array (5) shows 
that this is the truncated check number required for 
R,(1). This difficulty may be resolved by an exchange 
of zeros and ones in the (7 — 1) sequence, because the 
all-one number does occur. This leaves the check numbers 
for even errors unaltered, but exchanges zeros and ones 
in the odd-error check numbers. No additional restric- 
tions are imposed, however, because the all-parity check 
separates the two cases. Only conflicts in the (WZ — 1) 
sequence between runs of truncated check numbers for 
even errors and conflicts between odd errors restrict the 
code length. 

It is not now known how to predict @ przort which of 
the two methods, shift register of length m or all-parity- 
check and shift register of length (m — 1) produces 
longer codes; it depends entirely on the structure of the 
sequences. 


V. DECODING 


Decoding is particularly straightforward. Parity checks 
are performed on the received message in accordance 
with (3) and the resulting check number is stored. Check 
numbers are generated by a maximum length shift regi- 
ster in initial state R,(1). A counter, counting the shift 
pulses, is reset every time a new run of check numbers 
starts. When coincidence occurs between the received 
check number and the locally generated check number, 
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the number of resets of the counter identifies the error 
pattern, and the number in the counter identifies its 
position. The same procedure can be used with the all- 
parity-check codes, by running through the cycle twice, 
once for even numbers of errors and once for odd numbers. 

Because of the logical simplicity of the decoder, a more 
detailed decoder design is omitted. 


VI. REALIZABLE CoDES 


Abramson* has constructed codes for the correction 
of single and adjacent double errors. We are, therefore, 
interested in codes which correct errors in at least three 
adjacent positions. This means m > 6. All maximum 
length sequences generated by shift registers of length 
5, 6, 7, 8, and 9 have been investigated manually. As 
a result, codes were obtained of both types described 
above, m-place check numbers generated by m-stage 
shift registers and m-place check numbers generated by 
(m — 1) stage shift registers plus all-parity check. Table I 
lists the longest realizable codes together with their se- 
quence generators. Note that in each instance the reverse 
sequence generates a code of same length. Sequence 
generators for codes containing an all-parity check are 
underlined. (See Table I.) The Table can be extended 
without difficulty, preferably with the help of a digital 
computer. 


TABLE I 
REALIZABLE (n, k); CopES 


s =3 sa cao 

C n Cc n Cc n 

m =6 45 ll = ee ee So, 
m=7 Tata Soka ow ne tlie 

211 26 — = = = 

m = 8 313 40 211 15 — — 
Big oS Seay bade agape eet 

m=9 607 100 543 34 — — 

1423 ey 1131 38 2 — 

es 10 1055 206 1131 64 1021 19 
Note: Sequence generators for codes with all-parity check are 


underlined. 


The Table indicates that for each added check digit, 
for a given s, the maximum code word length is roughly 
doubled, as one would expect. It is not clear, however, 
which of the two described assignment methods is better 
in general. 

Finally, a word about the efficiency of the codes which 
might be measured in terms of the length of hypothetical 
complete codes as defined by the equality sign in (8) 
is in order. Given m and s, complete codes would permit 
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word lengths ne — 2+ s, which follows fro: 
3, the tabulated codes have values n/1 
ranging from 61 to 88 per cent and for s = 4, 48 to é 
per cent. For any value n/nc > 4, no code can exi 
which will allow the tabulated word length and wi 
correct clusters up to s, but has less redundant digi 
than the listed code. Most codes in columns s = 3 an 
s = 4 meet this criterion. Since only shift registers u 
to length 9 were investigated, the single code (19, 9 
which was obtained, tells little about s > 4. 


= els 


VII. EXAMPLE 


The following example of the construction of a coc 
will illustrate the method. The first code listed in Table 
is 11 digits along with six check digits. The sequenc 
generator is listed as 45 — (100101), a 5-stage shi 
register. The sum of the third and fifth stage are fe 
back. We start the sequence with 5 ones, which for tk 
truncated check numbers are then changed into fiy 
zeros. The (m — 1) sequence and the position of tk 
truncated check numbers of error patterns ending 1 
position 11 are shown below. (Error patterns are show 
with each number. ) 


LILILTOOOLLTOLIIOLTOINUOCOI OCA 
SS ee ee SS 
(11) 


(1) (101) (111) 

There are 2°-' = 4 distinct runs of check number 
Since the code has an all-parity check, only confli¢ 
between even numbers of errors and conflicts betwee 
odd numbers of errors have to be considered. Note ths 
R,(1) shifted by 11 positions is in conflict with the chee 
number for triple errors. Thus, the maximum code lengt 
is n = 11. The two runs for double errors (11) and (10! 
start 13 positions apart, which is no further restrictio1 
Starting with the truncated check numbers shown i 
the sequence above, we may obtain 2 different code 
of same length by continuing the check number rur 
either to the right or to the left. Fig. 2, shows the chec 
rules (a;;) for the (11, 5); code obtained by continuin 
the check number runs to the left. 


Inf. Digits Check Digits 


Fig. 2—Check rules for (11, 5)3 code. 


The code is capable of correcting any errors whic 
occur in any 3 adjacent positions. The construction | 
longer codes and for s > 3 is equivalent. 
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VIII. ApPLicaTIoNn 


Consider the application of (n, k), codes to a common 
rm of digital communication, namely teletype. The 
lestion is the maximum duration of a noise burst with 
hich a code can cope successfully. The information 
in groups of five digits. Let us measure time in units 
length of the original information digits (for example 
3.4 msec at a nominal speed of 60 wpm). Assume that 
le receiver will receive a digit correctly if normal recep- 
on takes place during at least half the duration of the 
git. This assumption, although somewhat arbitrary, is 
asonable in many practical cases. Since the noise burst, 
znal fade, or whatever else causes the disturbance 
scurs at random times with respect to a digit, a disturb- 
xce of duration between one-half and one and one- 
clf times a digit duration may cause a high error prob- 
vility for a single digit position. Also, no disturbance 
' length less than one digit length can corrupt two 
fjacent digits. Hence a code, which is capable of cor- 
cting errors in s adjacent position can cope with any 
sturbance up to a duration of s digits no matter what 
ie relative timing is. 

‘Consider first the encipherment of individual teletype 
2zaracters. To be realistic, we must assume that we 
ish to maintain a constant information rate. As a con- 
quence, the digit durations after encoding are not unity 
for uncoded transmission but k/n, and the permissible 
arst length is sk/n. Table II lists the maximum per- 
Hissible duration of a disturbance for the codes described 
this paper, together with corresponding numbers for 
me codes, which have been published previously. 


TABLE II 
PERMISSIBLE DuRATION 7’ oF DISTURBANCE 


| Type of Code i Remarks 
code ® 0.5 

amming! 9 0.55 Detects double errors 
bramson> 10 1.00 Corrects adjacent double errors 
hed? 16 0.94 Correct all triple errors 
reen-Sans Soucie! 15 1.00 
Hl 36 s=3 

13 1.54 s=4 

iy be 3s = 5 


\If disturbances do not occur too frequently, it is ad- 
|ntageous to encode a group of several teletype characters 
ito one block. Table I suggests these possibilities: 


Code Disturbance length 


11) 2characters (18, 10), 222 
12) 5characters (84, 25), 2.94 
13) 10 characters (60, 50), S230 
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Thus, at a nominal 60-wpm speed, a code encoding, 
for example, five teletype characters into one block of 
length 34, can handle disturbances of duration up to 
about 100 msec, provided such disturbances do not occur 
more often than about once per second. In the process, 
errors in up to four adjacent positions are corrected. 
We might also note that a code for five teletype characters 
capable of correcting up to all quadruple errors must 
have at least 18 check digits, instead of nine in our (34, 25), 
code, and the maximum disturbance length would be 
reduced to 78 msec. 


APPENDIX: DETERMINATION OF MAxiImumM CopE LENGTH 


Assume for the moment that the run of consecutive 
numbers #,(1) starts with the all-one number in the M 
sequence and call this assignment R,(1)*. For convenience, 
we take R#,(1)* = (111..11) and to a single error in position 
nm — 1 we assign the next number of the MW sequence 
which has ones in the first (m — 1) places and a zero 
in the last place. The first number of the run R,(11)* 
has then zeros in the first (m — 1) places and a one in 
the last place, because R,(11) = R,(1) + R,-1(1). In 
general R,(1 --- 1)* has zeros in the first (m + 1 — h) 
places if the associated error pattern contains an even 
number of errors. Conversely, for odd numbers of errors 
R,( --- 1)* starts with (m + 1 — A) ones. It follows 
that the numbers R,(1 --- 1)* are in those positions of 
the M sequence where the runs of zeros and ones, described 
in property 4 of m sequences (see Section IV) represent 
the first (m + 1 — h) places of corresponding m-place 
binary numbers of the M sequence. 

There are 2” runs of length (m — u — 1) of ones or 
zeros, and 2”~” runs of check numbers, 2 < h < s. Putting 
u = h — 2, we see that each run of (m + 1 — h) ones 
or zeros represents the first (m + 1 — h) places of a run - 
of R,(1 --- 1)*. Thus, in order to determine n for an 
(n, k), code, it is only necessary to inspect in the m- 
sequence runs of zeros and ones longer than (m — s) 
and count the number of shifts between these runs. Let 
n’ be the number of shifts between two such runs. That 
is, with 2 < h < s and 2 < h’ < s, assume that the 
first zero or one of a run of (m + 1 — A) zeros or ones 
in the m sequence, when shifted by n’ positions, coincides 
with the first zero or one of a run of length (m + 1 — h’), 
A run of length (m + 1 — h) belongs to an error pattern 
of length h, which has (n + 1 — h) check numbers. 
Therefore, (n + 1 — h) = n4;, orn = Noi th — 1. 
When there are several numbers n’ nearly equal with 
different h, this must clearly be refined to read n = 


(n’ + ave a ile 
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A Class of Codes for Signaling on a Noisy 


Continuous Channel” 


JOHNS LAKELEY JR, 


Summary—A class of codes for continuous channels is described. 
These are block codes in which the code words can be computed 
from a much smaller set of generators. It is shown that codes of 
this type exist which will yield arbitrarily small error rates at any 
signaling rate below channel capacity. In fact, if the generators 
are chosen at random, it is shown that the expected error rate 
obeys a bound established by Shannon for general random codes. 


INTRODUCTION 


‘YT is well known that, if a block code (continuous 
| or discrete) is generated by choosing the code words 

at random with the proper distribution and if the 
signaling rate is kept below channel capacity, the expected 
probability of error will approach zero with increasing 
code length.’ It is likely, however, that codes generated 
in this manner would have so little structure that the 
entire list of code words would have to be stored at the 
transmitter. It would be desirable for the transmitter 
to be able to compute the code words from some sort of 
algorithm. 

A class of codes called ‘‘Additive codes” is described 
below in which 2”” code words of length N are computed 
from NR + 1 generators, where #& is the signaling rate 
in bits per unit time. These additive codes for continuous 
channels are somewhat analogous to group codes for 
binary channels in which 2”” code words are computed 
from NR generators in a linear fashion. It will be shown 
‘that if the generators for an additive code are chosen 
at random, the expected probability of error obeys a 
bound given by Shannon’ for general random codes. 
Also the probability is high for large N that the code 
words will meet the channel constraints. 


NovraTION AND FORMALISM 


We avoid mathematical difficulties by considering 
channels whose inputs and outputs are sequences of real 
numbers occurring at discrete times. With certain reser- 
vations, each sequence may be thought of as the sample 
values of a band-limited time function. For N fixed but 
arbitrary, let 


Aes Ra op ts Aen) 5) Yaa aaa eer) 


where the X, and Y, are random variables representing 
the input and output samples, respectively, of a noisy 


* Manuscript received by the PGIT, October 19, 1959. 

+ Bell Telephone Labs., Inc., Murray Hill, N. J. 

1C, EK. Shannon, “A mathematical theory of communication,” 
Bell Sys. Tech. J., vol. 27, pp. 379-423, 623-656; July/October, 1948. 

2C. EK. Shannon, “Certain results in coding theory for noisy 
erence Information and Control, vol. 1, pp. 6-25; September, 
1957. 


communication channel. X and Y are thus rando: 


vectors. Let 


a= (2, °:- ty YY = (Yn) 


where the x, and y, are real numbers. In general, low 
case letters will stand for numbers and capitals for randor 
variables. The channel noise is described by a set « 
conditional distributions (one for each N) of Y cond 
tional on X. We will consider only those cases whe! 
the noise is represented by density functions P(y/s 
such that: 


probability [Ye A/X = 2] = / P(y/x) dy. 
A 


We state the channel constraint in the following fort 
to cover cases where the input stochastic process is nor 
stationary (as will be the case with block codes) 


Lim 7 > 1%) < fase ( 


with probability one where f(-) is a positive functio 
and fmax is a number. For example, if f(X) = X’, w 
have an average power limitation. 

By a ‘code of length N and signaling rate R”’ is mear 
m =—2)" we 


m7 


a set of 2”” labeled vectors v' --+ 2”, 


pri @ MU are rey ic 


The x’ are called code words. 

By a source is meant a method of assigning probabil 
ties to the stochastic process, X. We will consider tw 
sources. One, called the “test source,” is described by 
density function P(x) with 


probability [Xe A] = / Po) aes 
A 
The other, called the “coded source,’ may be thougl 


of as the result of coding the output of a discrete sourt 
with MZ symbols into the M code words. Thus | 


probability [X40 = x)= -M. 


All other vectors have probability zero. 


The probability of error P, for a given code is define 
by: 


| 
| 


il M < 
P. =a Lf, Put) ay 


where A, = {y | P(y/x‘) > P(y/zx") for some s ¥ 
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For a given test source and channel, we can define 
1e mutual information function I(x, y) by 


A Ys 


T(z, y) = + logs Se P(x)Q(a) 


here 


P(e, y) = P@)PQ/2), Qe) = [ P@, wae. 


| By plugging in the random variables, X and Y, we 
an get a random variable J = J(X, Y). The expected 
alue of J, (I), is of course the rate of transmission of 
iformation, and the maximum value of (/) over-all test 
ources (for large N if the channel has memory) is the 
annel capacity. As in Shannon’s work,” the bound on 
ae error probability will be stated in terms of the distri- 
ution function of J, 


p(a) = probability [I < a]. 


ADDITIVE CopDEs 


If a and b are numbers in the interval (0, 1), define 
+ bas their mod 1 sum; that is 


epb=a+t if ato<1 
Ey  Aersat g, he 


imilarly, if w and v are vectors with components in the 
terval (0, 1) 


u®Dv = (m1 On, -: v Dow). 


et 2, ® be vectors whose components are in the 
aterval (0, 1). Form 2” combination of the z’s as follows 


"=2Q@az Qaze+-:-:- fase. 


here the a, is the fifth binary digit in the binary expan- 
ion of the integer 7 — 1. Now we form our code words 
y applying a certain vector transformation (to be ex- 
Jained below) to the w’ 


NR NR 
Anre j; a 4 


= g(u’). 


‘ code formed in the above manner will be called an 
additive code.” The Z’s are called the ‘generators’ 
f the code. Note that the whole procedure becomes 
ointless if the vector transformation g(-) is too compli- 
ated. For most channels this is not the case, however. 
n fact, we will see that for oe channels, the 
ransformation becomes a simple “stretching” function 
pplied to each component, 7.¢., 


= h(w%). 


Ranpom ADDITIVE CODES 


If the generators are chosen at random, we have a 
random additive code;’ mathematically we let the gene- 
ators be independent random vectors Z°, Z' --- Z*" 
ach uniformly distributed in the N cube, 0 < Z; < ¢, 
=0---NR,s =1--- N. This gives rise to random 
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vectors W" and X’. It is easily verified that the W’ 
are uniformly distributed in the N cube, 0 < W? < 1, 
r=1---2*",.s=1--- N, and are pairwise independent. 

The function g(-) is used to match the code to the 
test source; 7.e., g(-) is chosen*® so that for any set, A, 

DFO aDility. exc ed = / Pix) ae! 
A 

The X’ will of course also be pairwise independent. 

For a random code the probability of error becomes 
a random variable and we can speak of the expected 
probability of error (P.). We can also speak of the proba- 
bility that the random code words, X", meet certain 
conditions. 

When using the coded source on a channel, (1) will be 
met if: 


cq Bleak 


If our code matches a test source which meets (1), the 
probability will be high when J is large that our random 
code words meet (2). 

More precisely we have the following. 


i 
Va (2) 


Theorem 1 

If a random additive code matches an ergodic test 
source, P(u), which meets (1), then the probability ap- 
proaches one with increasing N that the code words 
meet (2), provided P(u) is such that (f(X7)) and (f?(X7)) 
exist. 
Proof 

By the strong law of large numbers, (f(X7)) < finax 
Let 

eX) ey 

and 


S=3 


+t BCR 


Now since the X{ are pairwise independent 


Se aS ee 


as NV increases. 


The theorem follows from Tchebycheff’s inequality, 
Theorem 2 


If a random additive code matches a test source which 
yields a distribution of mutual information p(a), then 


(P) < p(k + 6) +2" 
for any number 86. 
3 This is possible in a variety of ways. For example let w, = F(a1), 


we = F(2x2/x1), ws = F(x3/a1, t) ....when the F’s are cumulative 
distributions. This obviously defines the «’s as a function of the w’s. 
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The proof follows essentially the method used by 
Shannon’ and will be omitted here. The extension of 
Shannon’s proof to the continuous case is trivial. The 
only other change to note is that if the probability of 
confusing the transmitted code word with a particular 
incorrect word is \, then the total probability of error 
is <2”). Shannon’s random code words were independent 
and he used the inequality 


—v*>1— My. 


It is well known that, for a memory-free channel with 
constraint of the form (1), the channel capacity is obtained 
with a test source having independent symbols; 7.e., 


P(x) = p(x)p(x2) --* p(ty). 


The random variable J is then the average of independent 
random variable and by the weak law of large numbers 


pia) > 0 for a <C. 


Thus in Theorem 2, if R < C, set 9 < C — Rand (P.) > 0. 

Actually there is a much larger class of channels for 
which p(a) — 0 for a < C when p(a) is computed from 
the proper test source. This class includes channels with 
finite number of internal states along with many other 
more general ones, of course. Let us call such channels 
‘Tegular.”’ Then it is obvious that for any regular channel 
the expected probability of error can be made to approach 
zero if R < C. 


Theorem 3 


If a regular channel has a capacity C obtained with 
with an ergodic test source P(u) and if (f’) and (f) exist 
[calculated with P(u)], then for any e > 0, 6 > 0, and 
any R < C, there exists an N and a random additive 
code of length N and signaling rate R such that the 
probability exceeds 1 — e that simultaneously the con- 
straint (2) is met and the probability of error is less than 6. 


Proof 


The above remark show that the expected probability 
of error can be made arbitrarily small. Since the error 
probability is a positive quantity, this means that the 
probability can be made arbitrarily near one that the 
error probability is less 6. Choose N so large that the 
probability that P, < 6 is greater than 1 — ¢/2 and 
also so large that the probability of meeting 2 is greater 
than 1 — ¢/2; then the probability of failing either re- 
quirement is <e. 
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An Example 
Let 
P(y/2) = pyi/a1) +++ PYn/ an) 
where 
Le ae a 
/ 20 
That is, the channel is memory-free with additive, whit 


Gaussian noise of unity power. 
Let 


P(Yi/%1) = 


f(x) = aj Viiax = 3. 
The capacity of this channel is given by: 
C=tlogi+3) =1 


It is obtained by a test source having independent symbol 
with 


bit/sample. 


Suppose we decide to signal over the channel at a rat 
of 0.75 bit per sample. Further assume that N = 10( 
turns out to be large enough to satisfy the various applica 
tions of the law of large numbers needed in the precedin; 
theorem. Then with R = 0.75, N = 100 we need 7 
generators of 100 components each. We therefore choos 
7600 independent samples uniformly distributed in the 
interval (0, 1), 23, r = @--- 75, r = 1, --+ 100, Thee 
7600 numbers are stored permanently at the transmitte: 
and receiver. When the transmitter wishes to send : 
block of 75 bits, it multiplies each of the 2” r = 1, --- 7 
by the appropriate binary digit and adds the 76 vector: 
mod one, to form a single vector of 100 components 


Each component, w, s = 1 --- 100 is “stretched” by 
letting 
x, = h(w,) 
where h(-) is the inverse of the function h7‘(-), 
aif * 1 —27/6 
h'(u) = —e dx. 


This gives the u, a Gaussian distribution with the corree 
power. They are then transmitted to the receiver. 

Unfortunately, there seems to be no simple detectio1 
procedure. Apparently the receiver must compute eacl 
of the 2” possible transmitted combinations and compar 
them with the received signal. It is possible, howevei 
that the structure of these codes will be of some hel 
in designing a systematic detection procedure. 
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A Relation Between Zero-Crossings 
and Fourier Coefficients for Band- 
width-Limited Functions* 

In a recent paper,’ an interpretation of 
a theorem by Titchmarsh? was discussed, 


showing that a function f(z) whose Fourier 
limited to the range 
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A special case of interest is 
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which is the well-known product expansion 
for sin nz. This function has uniformly- 
spaced zeros at « = +ka/n. By arranging 
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at intervals of a/n, the finite product 
expansion 
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is obtained heuristically. 
The identity is easily vertified by showing 
that the finite product expansion: 
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2) is an odd function of x; and 

3) contains no frequency component 
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The constant 2” is determined by ex- 
panding pairs of trigonometric products 
into sum and difference terms. 

The result may be used to derive addi- 
tional identities of interest as follows: 


log sin nx = (n — 1) log 2 


+ y log sin (x ie tr (4) 


eee: kar 
Couns = — cot {a + — (5) 
Teta n 
n=l x 
esc nz = ED csc” (: + te | (6) 
NM K=0 nN 
ua 
. (1 + 2k) 1 
IT sin on = 9 (7) 
ae 6.) ka n 
Oe eo! (n even) (8) 
(n-1) /2 Le ker n 
ul sin’ 7 = pi (n odd). (9) 


The finite product expansion can be 
generalized to represent a periodic band- 
width limited function containing exactly 
n (real) zero crossings in the interval 
(0, 27). Thus, let 


f(t.) = 0; 
< XE <e eens: 


(EES wee 


Ce oe (10) 
Then, except for an arbitrary scale factor, 
one may write 


f(x) = TT eos ie +o) a 
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where 

Pk at in) Sa a 
where the sine terms are replaced by 
cosine for subsequent mathematical con- 
venience. A Fourier series representation 


for f(x) in terms of its zero crossings may 
be obtained by expanding the finite product. 


gtr {we (52 +38) 
+ > cos (E = 1 |r + 46 — 6.) 
+£ho(5-2} 


{(@) = 


(12) 


The amplitude and phase of each Fourier 
component is calculated by the summation 
of unit vectors indicated. For n odd, the 
function is repeated negatively in the range 
2m to 47, and the final term is the funda- 
mental frequency component in the form 
A cos (x/2 + @). For n even, the final term 
is a constant and must be divided by two 
to correctly give the de component. In 
either case, n + 1 phase and amplitude 
terms are present. However, the amplitude 
of the first term is independent of the lo- 
cations of the n zero crossings. 

The product expansion can be further 
generalized to include complex conjugate 
zeros in addition to real zero crossings. 
If the factors are considered in pairs 


cos 2(« + ox) cos 3(" + di-1) 


= {cos ( k = tt) 
ach (« se E +o} 


- ole [25] 


Note that this factor vanishes at two values 
of x in (0, 27) and is specified in terms of 
the position of the mean of the zeros and 
half the distance between them. Now if 


xz is generalized to the complex plane 
z2=2+ jy, and zeros occur at 24 = X + jye 
and 21 = 2% — jyx then 


Gos 3(x + ¢,) cos Fh a Oren) 


= 4 (cosh Yy, — cos [x — rx]) (14) 


in*place of (13) for each pair of complex 
Zeros. 


F. E. Bonn 

C. R. CAHN 

Ramo-W ooldridge 
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School of Elec. Engrg. 
Purdue University 
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The Effect of a Limiter upon Signals 
in the Presence of Noise* 


It has been shown! (the proof requires 
only a few lines) that when the input to a 
band-pass limiter is a narrow-band signal 
A(t)/@(t) plus narrow-band Gaussian noise 
of mean-square value o? (both whithin the 
pass band of the device), the signal com- 
ponent of the otuput is 


(kA/o),F (3; 2; — A*/20°) /0(), 


where k is a constant [k = a/(27)!/2 if the 
limiter output is a or O accordingly as its 
input is positive or negative] and ,/; is 
the confluent hypergeometric function. 
This result is obtained from (4) of 
Blachman! by setting n = 0 (the case of 
the ideal limiter) and m = 1 (the case of 
fundamental-frequency operation). Thus, 
the noisy band-pass limiter simply acts 
like a nonlinear device which changes ampli- 
tude A into (kA/c) 1F'1(4; 2; —A?/2o?). 

Now we may determine what nonlinear 
device of odd symmetry has this effect on 
amplitudes. Representing its characteristic 
by f(x), we have 


(kA/o),F,(G; 2; —A’/2o°) 


=e a f(A cos ¢) cos ¢ dd/r. 


Expanding both sides of this equation as 
power series in A and equating coefficients, 
we find that 


f(a) = (ka/c),F 13; 3/2; —x”/20°) 
= (2n)'7k(—4 + erf x/o), 


* Received by the PGIT, October 20, 1959. 

1N. M. Blachman, ‘‘The output signal- to-noise 
ratio of a power-law device,” J. Appl. Phys., vol. 24, 
pp. 783-785; June, 1953, In ®, Io — i should be 
changed in both places to Jo + Ih. 

2 Janis Galejs, ‘‘Signal-to-noise ratios in smooth 
limiters,” Ire TRANS. ON INFORMATION THEORY, 
vol. IT-5, pp. 79-85; June, 1959. 
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where erf x = ie exp —3@ dt/(2x)!/ 
Hence, the effect of the limiter upon signals 
within its pass band in the presence 0 
noise is the same as that of an error. 
function-shaped nonlinearity in the absence 
of noise. It is for this reason that the 
action of an imperfect band-pass limite: 
with an error-function-shaped character. 
istic is especially easy to analyze;? its 
action is the same as that of an ideal band- 
pass limiter to whose input enough noise 
is added to obtain effectively the given 
error-function-shaped characteristic. 

As a check on the foregoing results, we 
may rederive it ab initio. If the input to 
an ideal limiter is the sum of a signal 2(?) 
plus Gaussian noise n(t) of mean-square 
value o?, both lying within its pass band, 
the output of the limiter before filtering 
being 0 or 1, according to the sign of the 
input, the instantaneous signal output 
voltage before filtering is simply the prob- 
ability that z + n is positive, viz., erf r/c. 
(The extra —4 above represents only a de 
and would not be passed by the filter.) 
Hence, the signal output (including dis- 
tortion) of the band-pass limiter is found 
by regarding the signal as passing through 
an error-function-shaped nonlinearity and 
then through a band-pass filter. To find 
the mean-square noise output, it is neces- 
sary to use the methods? of Blachman.! 


Newtson M. BLacHMAN 
Office of Naval Research 
U.S. Embassy 

London, Eng. 


3.N. M. Blachman, ‘On Manasse, Price, and 
Lerner,’” IRE Trans. ON INFORMATION THEORY, 
vol. IT-4, p. 174; December, 1958. 


On Best Approximation of Random 
Processes ef al.* 


The purpose of this note is to point out 
that a theorem of E. Schmidt,! on the best 
Lz approximation of a function of two 
variables by means of finite sums of products 
of functions of the first variable by functions 
of the second variable, can be extended 
rather simply to yield results on optimal 
signal representation? and coding for corre- 
lated signals.$ 


* Received by, the PGIT, August 25, 1959. 
1K. Schmidt, ‘Zur Theorie der linearen und nicht 
linearen Integralgleichungen,” Math. Ann., vol. 63, 
pp. 433-476; 1907. 
2 ramer, ‘‘Best finite linear approximatior 
to a second-order stochastic process,” Notices Amer. 
Math. Soc., vol. 5, p. 201; 1958. 
3H. P. Kramer and M. V. Mathews, ‘‘A lineas 
coding for transmitting a set of correlated signals,’ 
IRE Trans. on Inrormation THrEoRY, vol. IT, 
pp. 41-46; September, 1956. 
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For the sake of completeness, the expan- 
sion theorem and the approximation 
theorem shall be stated in the required 
generality. However, since their proofs, as 
presented for example in Smithies,* require 
no further elaboration than a reinterpreta- 
tion in the more general context, they are 
omitted. The reinterpretation consists of 
allowing the first variable of a kernel to 
assume values in one space and the second 
variable to do so in a completely different 
space. 

Let 91, Q: be two finite measure spaces 
with the respective measures pw, and po. 
Let Lp(u1), Le(u2) be the corresponding 
spaces of square-integrable functions. Let 
K(s, t), with s « Qi, t « Qs, satisfy the fol- 
lowing hypotheses. 

a) K(s, t) is measurable on Q; X Q2 and 


5 A | K(s, 0 |? du Xm)(s, 2) << @, 
(1) 


b) For each value of s, K(s, t) is a 
measurable function of ¢ such that 


[Ke OP mO< ©. @ 


c) For each value of ¢, K(s, t) is a measur- 
able function of s such that 


[ [KG OP mo<2. @ 


d) K(s, #) is not equivalent to the null 
function. 
Under the above hypotheses it can be 
shown that there exists a singular system 
(Un, Un; An) Of K defined by: 


u,(8) = 


df KG, Des) duals) (4) 


v(t) 


With respect to the singular system 
(Un, Un} An), in Which wu, is normal in Lo(p1) 
and v, is normal in Lo(u2) and Ax > Ani, 
the following LZ, expansion holds: 


K¢, 9 = Sono 


n=1 


=1,/ K(s, tun(s) duy(s). (5) 


(Ls). (6) 


Moreover the following approximation 
theorem is valid. 

Theorem (1. Schmidt!): Let (tn, Un; Xn) 
be a singular system for the ZL, kernel K 
and write 


Ko, ) = Be. @ 


v=1 


If ai, --* 5 Gn, 61, °°: , by are arbitrary 
Lo(u1), Lo(u2) functions, then 


” 


Cambridge 


4¥. Smithies, “Integral Equations, 
147-150; 


University Press, Cambridge, Eng., pp. 
1958. 
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[1X69 - Laws P 


¥ 4X Qe 


D(X ue) (Ss, t) 
> i, | Ko, ) — KG, oP 
1X Qe 
» (uy X ps) 


= a 8) 


v=n+1 


First Application 


Let x(w, t) be a second-order random 
function defined on the finite interval J 
of the real time axis. Upon identifying Q; 
with ©, the sample space of the process, 
and yi with P the associated probability 
measure, and upon identifying Q2 with I 
and ype with ordinary Lebesgue measure, 
one sees that (1) represents the Loéve- 
Karhunen expansion of a second-order 
process and the approximation theorem 
implies the following. 

Theorem? Let x(t) be a mean-square 
continuous stochastic process on J with 
zero mean. Let n be a positive integer. Let 
a1, 2, , Q be n arbitrary random 
variables and gi(t) -:: , ¢,(t) n arbitrary 
functions. Then 


[z 


Te 2) ae, 


|V 
—_ 
by 


=e (9 


where £ are the random coefficients, and 
y(t) the time functions appearing in the 
Loéve-Karhunen expans‘on.® That is, 


a) d, | rs, ) ¥,() dé = (3), (10) 


b) & => | el) vO au, (11) 


(12) 


and 


@) Bie} = 8s, 


a) f vl) WO at = al. (13) 


Here r(s, ¢) is the covariance function of 
x(t). 

It should be pointed out that this result 
is more general than a similar one presented 
by K. L. Jordan, Jr.,° in that the random 


5 M. Loéve, ‘‘Probability Theory,’ D. Van No- 
Buran Co., Inc. ., New York, N. Y., p. 464; 1955. 
AGE Jordan, Spray IME L'T. Res. Lab. Electronics, 
Caukaie Mass., Quart. Progr. Rept. No, 52; 
January, 1959, 
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variables a4, * , GQ, are arbitrary here 
whereas in Jordan® they are assumed to 
be the Fourier coefficients with respect to 
the system of functions ¢i(t) -:- , gn(t). 
The above theorem suggests an optimal 
pulse amplitude modulation scheme. If 
it is desired for purposes of transmission 
to represent a time limited signal from a 
certain ensemble by means of a fixed finite 
set of numbers, then the best choice con- 
sists of the integrals of the signals against 
the singular functions {y,(¢)}. Having 
transmitted these numbers, an approximant 
to the original signal can then be recon- 
structed at the receiver by using the 
received numbers for coefficients in the 
truncated Loéve-Karhunen expansion. 


Second Application 


Let ©; be as in the first application, but 
identify Q2 with a finite or infinite set of 
integers and mw: with counting measure. 
Then one has the following theorem, 

Theorem: Let 4(w) = {x1(w), to(w), -°* , 
Xm(w)} be a random vector. If ai, a2, +++ , Gn 
is a sequence of random variables and 
Or, , 6” n arbitrary m-dimensional 
vectors, then 


where 
s=1 


and R;; is the covariance matrix of z. 


CoNcLUSION 


A random process can be regarded as a 
function of the two variables, time and 
position in phase space. On. this premise, 
a theorem by HW. Schmidt on the best 
least-squares ‘‘diagonal’’ representation of 
a function of two variables becomes appli- 
cable to random processes. The result is 
that, for a given random process x(t), of 
all approximants having the form 


N 
Dd angar(t), 
1 


with N arbitrary random variables a, and 
N arbitrary time functions ¢,(¢), the 
Loéve-Karhunen representation is optimal. 
It is important to note the fact that it is 


“not necessary to assume a priori that the 


a’s are the Fourier coefficients correspond- 
ing to the ¢’s. 


Henry P. KRAMER 
TEMPO 

General Electric Co. 
Santa Barbara, Calif, 
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On the Cross Correlation Between 
Two Noisy Channels* 


In this note we wish to derive a result 
for a two-channel configuration similar to 
one recently proved by Keilson, Mermin and 
Bello,! where mutually uncorrelated, Gaus- 
sian, narrow-band, channel inputs were 
assumed. Here, the bandwidth and cor- 
relation restrictions are removed and a 
more general nonlinear device is considered; 
specifically, the device is such that the 
output, v(t), is given in terms of the input, 
u(t), by the relationship: 


v(t) = fe Klu(é — 7), 7] dr, (1) 


a) 


where K(z, y) is a real-valued function of 
its arguments satisfying the one-sided 
homogeneity condition, 


K(de, y) = VK(a, y) (2) 


for all \ > 0 and some real » This class of 
devices is a subclass of Zadeh’s® class Ij 
and contains such well-known (instanteous) 
devices as the ideal limiter (vy = 0), full 
and half-wave nth law devices (vy = n) 
and any tandem conbination of the fore- 
going with a linear filter. 

Consider two channels, each containing 
a nonlinear device characterized by (1) 
and (2), the devices being identical for the 
two channels. Let the input to Channel 1 
be N.(t) + Ni(t) and the input to Channel 
2 be Nio(t) + N2(t), where the {N,(t)}, 
(« = 0, 1, 2) are mutually correlated, 
stationary, zero-mean Gaussian noises; the 
problem then is the determination of the 
output cross correlation’ 


vir) = (v(t)o(t + 7)). 


From (1) and the assumed input forms, 
vi) = ff (KIN - 9 


Sie Nit aaa é), £]-K[No(t Lag a n) 


+ NAb 7 — 4), 9) ab dy. (8) 
Letting 


ty = Not nt) Sev RL =" e) 


and 
to = N (t+ 7 — 7) 
= Ni(t ic re n) 


the joint probability density of 2, and 22 
is* P(x1, 2; a, g(r =) a f), B), where 


* Received by the PGIT, September 21, 1959. 

1J. Keilson, N. D. Mermin, and P. Bello, “A 
theorem on cross correlation between noisy channels,” 
IRE Trans. on InrormMation THEORY, vol. IT-5, 
pp. 77-79; June, 1959. 

27, A. Zadeh, ‘Optimum nonlinear filters,’ J. 
Appl. Phys., vol. 24, pp. 396-404; April, 1953. 

’ Angle brackets denote ensemble averaging. 

4J. H. Laning and H. Battin, ‘‘Random 
Processes in Automatie Control,” McGraw-Hil, 
ath Co., Inc., New York, N. Y., p. 78 and p. 156; 
1956. 


er! 
Py: Geese 2 SS 
at ) IWrrv/ac — b° 
ca, — 2bx,x%, + a] 


a = (ri()), 6B = (x2(0) 


and 


G(r) = (x,(t)x2(t + 7)). 


From (4), it is apparent that P(x, 2»; 
a, op(7), 0?) = Wai, x2; 0%, p(r)) is the 
usual second-order probability density for 
a zero-mean, stationary Gaussian process 
with rms o and normalized autocorrelation 
function p(7). Further, the identity 


P(avV/a, yV/B; a, aBh(7), 8) 


— We, y; 1, h(x) 
Va 


is easily verified. Eq. (3) may be rewritten 


wo [ELL Latent 


K(x, ) P(e, » v2 >a, 


(5) 


-g(r — n + £&), B) da, ae fa dn. (6) 


Under the change of variable z = x,/ V/. im : 
y = 2/~/ and the definition 


h(r) = ge 
abe V ap 


(6) becomes 


VY; 2(7) = (a) fe 


ts | [ [. Ke@oxen 


Wie, Y; Le 


“fir mire) eae ay | d&dn, (7) 


where use has been made of (5) and relation 
(2). If we define ¢,(c, +) to be the output 
autocorrelation function of a single non- 
linear device with characteristic K(2, y) 
for a stationary, zero-mean, Gaussian input 
having rms o and normalized autocorrela- 
tion function p(r), then the quadruple 
integral in (16) is equivalent to the quantity 
or(1, 7). Thus 


Wi2(7) =z (8), (1, i) (8) 


which is the main result of this note. 

For the subclass of zero-memory devices 
satisfying (1) and (2) and for mutually 
uncorrelated N,(t) (¢ = 0, 1, 2), (8) can be 
shown to reduce to 


Vial) =Fbanlo) 


March 


where oo and poo are respectively the rms 
and normalized aurocorrelation function 
for the process No(¢), and 


2 Z —-1/2 
Viaee {(i oe Ne +4 “)t 
Jo 00 


Bq. (9) is a result which is formally analo- 
gous to (12) of the previously mentioned 
work, 

Two of the explicit examples cited 
previously! are included in the formulation 
of (8). The square-law detector may be 
obtained from the tandem combination of 
an instantaneous square device and low- 
pass (linear) filter while the envelope 
detector is obtained by a half-wave linear 
device followed by a low-pass filter. 

From the proof of (8), it is apparent 
that the result may be generalized to 
include the case’ of P additive zero-mean, 
stationary, Gaussian noises in Channel 1 
and Q such inputs in Channel 2 with 
arbitrarily given correlations existing among 
the various noises. The result will again 
have the form of (8); however, obvious 
modifications must be made in the defini- 
tions of a, 8, and h(r). 


Joun L. Brown, JR. 
Ordnance Res. Lab. 
The Pennsylvania State University 
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Probability Distribution for the De- 
rivative of the Envelope of Signal and 
Gaussian Noise* 


In a recent paper, Brown! demonstrated 
that the first probability distribution for 
the derivative of the envelope of Gaussian 
noise is Gaussian. This note serves to 
point out further conditions under which this 
property prevails. If a constant amplitude, 
constant frequency ‘carrier’? signal is 
linearly added with the input Gaussian 
noise prior to the detector action, then 
the first probability distribution for the 
derivative of the envelope of such a process 
will be Gaussian, provided that the fre- 
quency spectrum of the input noise possesses 
narrow-band symmetry about the carrier 
frequency. This may be readily shown in 
the expression for the characteristic function 
whose development may be obtained in a 
simple, straightforward fashion as briefly 
reviewed in the following. 

If we express the signal applied to the 
input of the envelope device as 


e(t) = So Cos wot + a(£) cos wot 
+ y(t) sin wot (1) 


* Received by the PGIT, August 31, 1959. , 
eae 2 Brown, ‘Some results on noise through 
circuits, IRE Trans. on InrorMATION THEORY, 
vol, IT-5, pp. 217-227; May, 1959, 
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where S) = amplitude of carrier signal, 
x(t) = inphase noise component, and 
y(t) = quadrature noise component; then 
the envelope function becomes 


at) = V[x(t) + SP + yd). 


The derivative of z becomes 


(a + Sot + yy z 
V(x ie Sole ate y 


The characteristic function associated 
with the first probability density function 
of 2(¢) is given by 


v= [fff 


where 4; = x + So. 
If the input Gaussian noise is a stationary 
process, then it may be shown that 


He, 9, 07, = f(a;yiy, 2) ~~) 


where each f is a Gaussian probability 
density function. The following covariance 
matrices are associated therewith: 


(2) 


at) = (3) 


(4) 


os VO\s | o> —VO102 
and 
VO\02 1 —_Vd0\02 a; 
where 
2 1 z y fe J 
e = eal Wo!) deo 
2a —@o 
= (a, — 8) = yy" — (6a) 
Contributors 
Bernard Elspas (S’ 46—A’ 48—S’ 53 


A’ 55—M’ 56) was born in New York, 
N. Y., on July 26, 1925. He received the 
B.E.E. degree from 
the College of the 
City of New York in 
1946, the M.E.E. de- 
gree from New York 
University, New 
York, in 1948, and 
the Ph.D. degree 
from Stanford Uni- 
versity, Stanford, 
Calif., in 1955. 

From 1949 to 1951 


B. Evspas 


Correspondence 


2 Lace 2 
02 = On fee wo’ Ww’) da’ 


2 «2 


Seay (6b) 
V01}0_ = as ‘e wo’ Ww’) dw’ 
ria 7S ae 
= ty = —Y(x, — So). (6c) 
W(w) equals the spectral density of the 
input noise and w’ = w — wo. The joint 
distribution function (4, y, 21, y) ac- 
cordingly becomes 
f(a, Y, X, y) = aie : 2 
(2m) o;02(1 = ¥) 
1 E z 
exp eee 
21 — 7’) 
x, — So) Qa 
Bie (a f o) al 7] vey 
O71 O71 0102 
Qvy(x, ae 


80 |. (7) 


After substituting (7) into (4), we can 
integrate with respect to ¢ and y to obtain 


V,(Q) = {exp is o(1 = “| 
AL Lar 2 [a5] 


rapes 
-exp ce (r cos 6 
Oj 


0102 


— tyo,o.Qsin » | dé ar} (8) 
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where we have 
accordance with 


changed coordinates in 


%, =rcos@ and y=rsin @. 


Integrating with respect to 6 yields 


¥.(9) = exp |-£ oi(1 | 


oe} 2 2 
pe ree 
[ nex 2 | 


‘TlaVu — k) du (9) 


where 


==" ANdus == Nae 


SS 
O1 

Brown’s conclusions are demonstrated 
by letting So = 0 (and hence a = QO), to 
obtain for V;(w) the following: 


V.(Q) = exp | (lites ‘| (10) 


which is, as is well known, the characteristic 
function for a normal distribution of zero 
mean and variance o?(1 — v?). 

When the frequency spectrum of the 
input noise has symmetry about the carrier 
frequency, then by (6c), » = 0 (and hence 
k = 0). Accordingly, the integral expression 
(9) will not be a function of Q; it will, in 
fact, be unity for all ‘“a.’’ Therefore, for 
such a case 


W.(Q) = exp E |. (11) 
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F, Louis H. M. Stumpers (A’ 49—M’ 56— 
SM ’58) was born in Eindhoven, The 
Netherlands, on August 30, 1911. He 
joined the Philips Research Laboratories, 


March 


Hindhoven, in 1928, where he remained 
until 1934 when he left to study mathe- 
matics and physics at Utrecht Univer- 
sity until 1937. After 
his doctoral examina- 
tion and_ research 
work on Stark effect, 
he returned to the 
Philips Laboratories 
as a research phys- 
iscist in 1938. His 
first work was con- 
cerned with the 
application of semi- 
conductors in tele- 
communication. 
In 1939 he joined the 
Department of Fundamental Radio Re- 
search under Professor van der Pol. In this 
group he did work on circuit theory, cables, 
frequency modulation, stochastic problems, 
and noise. In 1946 he received the doctoral 
degree in technical science from Delft 
University. Since then he has done work 
on low-noise receivers, general noise prob- 
lems and information theory, and radio 
interference. In 1952-1953, on leave of 
absence from the Philips Laboratories, he 
was a research associate at the Research 
Laboratory of Electronics of the Massa- 
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chusetts Institute of Technology, Cam- 
bridge. 
From its conception in 1948, Dr. 


Stumpers bas been a member of the board 
of the Foundation for the Study of Radio- 
astronomy in The Netherlands. He is a 
member of the Dutch Physical Society, 
the Dutch Radio Society, and a recipient 
of the Veder Radio Prize. He is also chair- 
man of the committee for information 
theory of URSI, vice president of the. 
Netherlands CISPR Committee, and a 
member of the CISPR Steering Committee. 
He also attended the CCIR Conferences in 
Warsaw and Los Angeles as an expert for 
the Netherlands delegation. 
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